You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/06/11 02:19:15 UTC
[jira] Created: (CASSANDRA-1181) kinder gentler compaction
kinder gentler compaction
-------------------------
Key: CASSANDRA-1181
URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
Project: Cassandra
Issue Type: Task
Reporter: Jonathan Ellis
Assignee: Brandon Williams
Fix For: 0.7
I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901643#action_12901643 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
+1
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181-trunk.txt, 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901628#action_12901628 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
let's make priority=1 the default, and add the jvm options to cassandra-env.sh
should we reject setting compaction to higher priority than NORMAL? no idea what will happen if you do higher-than-normal on a JVM as non-root, since we've explicitly defeated its protection against doing that :)
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181-trunk.txt, 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1181) kinder gentler
compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882709#action_12882709 ]
Brandon Williams edited comment on CASSANDRA-1181 at 6/25/10 4:57 PM:
----------------------------------------------------------------------
There is some weirdness here. With the JVM options added, I see the compaction thread at niceness -5 as I would expect, however many other threads (like Concurrent Mark Sweep) are running between -3 and -5. If I remove the -XX:ThreadPriorityPolicy=42 option, even though I'm running as root, all threads run at a normal priority.
was (Author: brandon.williams):
+1
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-1181:
-----------------------------------------
Assignee: Jonathan Ellis (was: Brandon Williams)
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo updated CASSANDRA-1181:
---------------------------------------
Attachment: CompactionManager.java
How about using some information from tpstats to make this adaptive. See sample code?
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882612#action_12882612 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
to enable, add to cassandra.in.sh:
-XX:+UseThreadPriorities \
-XX:ThreadPriorityPolicy=42 \
-Dcassandra.compaction_priority=1 \
(TPP=42 is a workaround to allow us to *lower* thread priority without root privileges, explained at http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html)
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882722#action_12882722 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
To reiterate, this patch leaves the default behavior unchanged, but allows decreasing CM priority as described above.
if you don't have enough capacity to both compact and handle read/write load, then you're screwed. writing a manual scheduler that may or may not do slightly better in the short run is not going to change that.
what we want to do is slow compaction down so that instead of short violent bursts of CM activity you spread it out over a longer period of time.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881733#action_12881733 ]
Edward Capriolo commented on CASSANDRA-1181:
--------------------------------------------
I think ThreadPriority will help, however CPU Priority is not directly linked to IO scheduling. I believe between the the statistics cassandra keeps we could accomplish something adaptive. Also if we want a solution that is not 100% java pure we can read /proc/diskstats and use actual disk utilization to be adaptive.
All these methods I suggested fall apart if non-compaction traffic is thrashing the disk, but if that were the case the node was not healthy in the first place.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1181:
----------------------------------------
Attachment: 1181-trunk.txt
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181-trunk.txt, 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882724#action_12882724 ]
Brandon Williams commented on CASSANDRA-1181:
---------------------------------------------
+1
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1181:
--------------------------------------
Fix Version/s: 0.6.3
(was: 0.7)
Component/s: Core
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886068#action_12886068 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
Thanks for trying it out!
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1181:
--------------------------------------
Attachment: 1181.txt
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo updated CASSANDRA-1181:
---------------------------------------
Attachment: stats.txt
Just wanted to check back in. I enabled the thread priorities and initiated a compaction.
Both my latency and tpstats looked on par with other nodes in the cluster that were not compacting at the time. This looks great. I included some system statistics to show off. Thanks!
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1181) kinder gentler
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882612#action_12882612 ]
Jonathan Ellis edited comment on CASSANDRA-1181 at 6/25/10 12:09 PM:
---------------------------------------------------------------------
to enable, add to cassandra.in.sh:
-XX:+UseThreadPriorities \
-XX:ThreadPriorityPolicy=42 \
-Dcassandra.compaction.priority=1 \
(TPP=42 is a workaround to allow us to *lower* thread priority without root privileges, explained at http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html)
was (Author: jbellis):
to enable, add to cassandra.in.sh:
-XX:+UseThreadPriorities \
-XX:ThreadPriorityPolicy=42 \
-Dcassandra.compaction_priority=1 \
(TPP=42 is a workaround to allow us to *lower* thread priority without root privileges, explained at http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html)
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882719#action_12882719 ]
Stu Hood commented on CASSANDRA-1181:
-------------------------------------
I'm -0 on changing the compaction thread priority, as opposed to using a solution like ecapriolo has suggested. This solution might even backfire and cause worse read performance if compaction loses enough priority that you end up with tons of sstables waiting to be compacted.
While 'sleeping' compaction threads manually is akin to building our own scheduler, we have so much more information than the OS scheduler does that it will probably be worthwhile. The ideal situation would be that while writes are incoming, compaction is running almost constantly, so that it is fully amortized. That is, there should only be one possible cascaded compaction at a time, and it should finish immediately before the next compaction becomes possible.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reopened CASSANDRA-1181:
-----------------------------------------
Assignee: Brandon Williams (was: Jonathan Ellis)
Reopened for trunk. We need to add these JVM options to cassandra-env.sh and expose the compaction thread priority in the yaml config.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-1181.
-----------------------------------------
Reviewer: jbellis
Resolution: Fixed
Committed.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181-trunk.txt, 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1181:
----------------------------------------
Attachment: 1181-trunk.txt
Updated to make MIN_PRIORITY the default, and constrain values to be less than or equal to NORM_PRIORITY. Setting it any higher is silly anyway.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181-trunk.txt, 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1181:
----------------------------------------
Attachment: (was: 1181-trunk.txt)
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java, stats.txt
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882717#action_12882717 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
it's expected that everything runs at normal priority w/o TPP, even as root.
normally, you would set TPP=1 to allow thread priorities to be set in the jvm. the reason 42 is a trick is that the jvm will not let you set it to 1 if you are not running as root, but the only reason you have to be root as far as linux is concerned is to *raise* thread priority. lowering it is always ok. so setting it to 42 fools the "is tpp 0" check on setPriority, without having the jvm's excessively stringent santity-checking hose you.
source diving at http://stackoverflow.com/questions/1662185/do-linux-jvms-actually-implement-thread-priorities
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882709#action_12882709 ]
Brandon Williams commented on CASSANDRA-1181:
---------------------------------------------
+1
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.6.3
>
> Attachments: 1181.txt, CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1181) kinder gentler compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881730#action_12881730 ]
Jonathan Ellis commented on CASSANDRA-1181:
-------------------------------------------
using thread priority makes that unnecessary. that is why it's a better solution.
> kinder gentler compaction
> -------------------------
>
> Key: CASSANDRA-1181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1181
> Project: Cassandra
> Issue Type: Task
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: CompactionManager.java
>
>
> I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
> You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
> In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.