You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeremy Hanna (JIRA)" <ji...@apache.org> on 2019/06/13 18:29:10 UTC
[jira] [Updated] (CASSANDRA-11714) shutdown script should wait more
than 30 seconds before force kill
[ https://issues.apache.org/jira/browse/CASSANDRA-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-11714:
-------------------------------------
Complexity: Low Hanging Fruit
> shutdown script should wait more than 30 seconds before force kill
> ------------------------------------------------------------------
>
> Key: CASSANDRA-11714
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11714
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Tools
> Reporter: Wei Deng
> Assignee: Wei Deng
> Priority: Low
> Labels: lhf
>
> I'm running with 3.0.6 and it appears that if I let it run for multiple days (ingesting a never-ending streaming data from Kafka), then the next time I'm restarting the JVM by "service cassandra restart", it will take 9 minutes to replay the commit log segments (the commitlog_total_space_in_mb is using the default 8GB and I observed close to 8GB commitlog segment files before the restart). Here is a fragment of the system.log showing how long it takes:
> {noformat}
> INFO [main] 2016-05-04 19:06:54,356 CommitLog.java:168 - Replaying /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248425.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248429.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248433.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248437.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248441.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248445.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248449.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248453.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248457.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248461.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248466.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248470.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248474.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248478.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248482.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248486.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248490.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248494.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248498.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248502.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248506.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248510.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248514.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248518.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248522.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248526.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248528.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248529.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248530.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248531.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248532.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248533.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248534.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248535.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248536.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248537.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248538.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248539.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248540.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248541.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248542.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248543.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248544.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248545.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248546.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248547.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248548.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248549.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248550.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248551.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248552.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248553.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248554.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248555.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248556.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248557.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248558.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248559.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248560.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248561.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248562.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248563.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248564.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248565.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248566.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248567.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248568.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248569.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248570.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248571.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248572.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248573.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248574.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248575.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248576.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248577.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248578.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248579.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248580.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248581.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248582.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248583.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248584.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248585.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248586.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248587.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248588.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248589.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248590.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248591.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248592.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248593.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248594.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248595.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248596.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248597.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248598.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248599.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248600.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248601.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248602.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248603.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248604.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248605.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248606.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248607.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248608.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248609.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248610.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248611.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248612.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248613.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248614.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248615.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248616.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248617.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248618.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248619.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248620.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248621.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248622.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248623.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248624.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248625.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248626.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248627.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248628.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248629.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248630.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248631.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248632.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248633.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248634.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248635.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248636.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248637.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248638.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248639.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248640.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248641.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248642.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248643.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248644.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248645.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248646.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248647.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248648.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248649.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248650.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248651.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248652.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248653.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248654.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248655.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248656.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248657.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248658.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248659.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248660.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248661.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248662.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248663.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248664.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248665.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248666.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248667.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248668.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248669.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248670.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248671.log
> INFO [main] 2016-05-04 19:15:44,172 CommitLog.java:170 - Log replay complete, 331463 replayed mutations
> {noformat}
> It appears that the init script here only gives 30-second grace period after SIGTERM before forcefully killing it.
> https://github.com/apache/cassandra/blob/cassandra-3.0/debian/init/#L101
> If we could increase this 30-second to a higher wait time like 1 or 2 minutes we could make the startup delay much shorter.
> Keep in mind that we don't always have to wait for this long during the shutdown so we're not really losing anything with this improvement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org