You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (Created) (JIRA)" <ji...@apache.org> on 2011/10/20 17:37:11 UTC
[jira] [Created] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Config should default to a higher-throughput configuration for log.flush.interval
---------------------------------------------------------------------------------
Key: KAFKA-164
URL: https://issues.apache.org/jira/browse/KAFKA-164
Project: Kafka
Issue Type: Improvement
Components: config
Affects Versions: 0.7
Reporter: Jay Kreps
Assignee: Jay Kreps
Fix For: 0.7
Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Chris Burroughs (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131854#comment-13131854 ]
Chris Burroughs commented on KAFKA-164:
---------------------------------------
There are good things to be said about having the default configuration be the "overly paranoid do not loose your data" one, even though once they think about what that means no one actually wants that. I'd stick with round numbers and a note elaborating on the trade-offs.
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jay Kreps (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps resolved KAFKA-164.
-----------------------------
Resolution: Fixed
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164-v2.patch, KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133787#comment-13133787 ]
Jay Kreps commented on KAFKA-164:
---------------------------------
Err, ignore the random Jmx file in the patch...left over from another commit.
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164-v2.patch, KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jay Kreps (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-164:
----------------------------
Attachment: KAFKA-164.patch
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131743#comment-13131743 ]
Jun Rao commented on KAFKA-164:
-------------------------------
Maybe shrink flush time to 500ms?
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133826#comment-13133826 ]
Jun Rao commented on KAFKA-164:
-------------------------------
+1 on changes in server.properties
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164-v2.patch, KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-164) Config should default to a
higher-throughput configuration for log.flush.interval
Posted by "Jay Kreps (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-164:
----------------------------
Attachment: KAFKA-164-v2.patch
I added documentation on the tradeoffs for flush policy. I also went back and improved the documention across the board for the options we have and added in commented out stubs for options that were missing but are available (such as size-based retention policy).
I find no one ever reads the code to find out about configs so the config file is a pretty good place to document them.
> Config should default to a higher-throughput configuration for log.flush.interval
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-164
> URL: https://issues.apache.org/jira/browse/KAFKA-164
> Project: Kafka
> Issue Type: Improvement
> Components: config
> Affects Versions: 0.7
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.7
>
> Attachments: KAFKA-164-v2.patch, KAFKA-164.patch
>
>
> Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira