You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Patrick Hunt (Created) (JIRA)" <ji...@apache.org> on 2011/10/22 01:50:32 UTC
[jira] [Created] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
add logging/stats to identify fsync stalls
------------------------------------------
Key: ZOOKEEPER-1239
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
Project: ZooKeeper
Issue Type: Improvement
Components: server
Reporter: Patrick Hunt
We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149133#comment-13149133 ]
Hadoop QA commented on ZOOKEEPER-1239:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503499/ZOOKEEPER-1239_br34.patch
against trunk revision 1201045.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/789//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/789//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/789//console
This message is automatically generated.
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 3.3.4, 3.4.0, 3.5.0
>
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Camille Fournier (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150653#comment-13150653 ]
Camille Fournier commented on ZOOKEEPER-1239:
---------------------------------------------
Are you sure we should be doing this timing using system.nanotime?
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 3.3.4, 3.4.0, 3.5.0
>
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-1239:
------------------------------------
Attachment: ZOOKEEPER-1239_br33.patch
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150658#comment-13150658 ]
Mahadev konar commented on ZOOKEEPER-1239:
------------------------------------------
Camille,
Can you please commit this to 3.4 branch as well?
thanks!
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 3.3.4, 3.4.0, 3.5.0
>
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Patrick Hunt (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt reassigned ZOOKEEPER-1239:
---------------------------------------
Assignee: Patrick Hunt
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-1239:
------------------------------------
Attachment: ZOOKEEPER-1239_br34.patch
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134319#comment-13134319 ]
Patrick Hunt commented on ZOOKEEPER-1239:
-----------------------------------------
Todd pointed me at this commit in hbase which does similar (sync time tracking)
---
HBase does a log message when any sync() call takes longer than a second or so.
commit 96621d010f9a924e2af338a10b584b7f07caf17d
Author: Michael Stack <st...@apache.org>
Date: Mon May 11 17:32:24 2009 +0000
HBASE-1404 minor edit of regionserver logging messages
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Camille Fournier (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150657#comment-13150657 ]
Camille Fournier commented on ZOOKEEPER-1239:
---------------------------------------------
Eh, I guess the popular consensus has changed on using nanotime for this sort of thing, so disregard my question. I'll put this in shortly.
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 3.3.4, 3.4.0, 3.5.0
>
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1239) add logging/stats to identify
fsync stalls
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151136#comment-13151136 ]
Hudson commented on ZOOKEEPER-1239:
-----------------------------------
Integrated in ZooKeeper-trunk #1366 (See [https://builds.apache.org/job/ZooKeeper-trunk/1366/])
ZOOKEEPER-1239. add logging/stats to identify fsync stalls. (phunt via camille)
camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202360
Files :
* /zookeeper/trunk/CHANGES.txt
* /zookeeper/trunk/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml
* /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
> add logging/stats to identify fsync stalls
> ------------------------------------------
>
> Key: ZOOKEEPER-1239
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1239
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 3.3.4, 3.4.0, 3.5.0
>
> Attachments: ZOOKEEPER-1239_br33.patch, ZOOKEEPER-1239_br34.patch
>
>
> We don't have any logging to identify fsync stalls. It's a somewhat common occurrence (after gc/swap issues) when trying to diagnose pipeline stalls - where outstanding requests start piling up and operational latency increases.
> We should have some sort of logging around this. e.g. if the fsync time exceeds some limit then log a warning, something like that.
> It would also be useful to publish "stat" information related to this. min/avg/max latency for fsync.
> This should also be exposed through JMX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira