You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Yang Ye (Created) (JIRA)" <ji...@apache.org> on 2012/02/09 20:04:00 UTC
[jira] [Created] (KAFKA-268) Add
Add
----
Key: KAFKA-268
URL: https://issues.apache.org/jira/browse/KAFKA-268
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.7
Reporter: Yang Ye
Priority: Minor
Add another reconnect threshold to the syncProducer: the elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210620#comment-13210620 ]
Jun Rao commented on KAFKA-268:
-------------------------------
How about the following?
private var lastConnectionTime = System.currentTimeMillis - SyncProducer.randomGenerator.nextDouble() * config.reconnectInterval
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, random_last_connection_time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210382#comment-13210382 ]
Jun Rao commented on KAFKA-268:
-------------------------------
A suggestion: It's possible that we have a number of producer clients all started at around the same time. Then they will all reconnect at almost exactly the same time. This is probably not good for load balancing. What we want is to spread the reconnects. One way to do that is to initialize lastConnectionTime to a random timestamp btw (now - reconnectTimeInterval) and now.
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210633#comment-13210633 ]
Yang Ye commented on KAFKA-268:
-------------------------------
This is clearer, thanks. This part is also included in the shallow_iterator patch.
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, random_last_connection_time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-268) Add another reconnection condition to
the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-268:
--------------------------
Attachment: time_based_reconnect_feature.patch
As suggested by the first comment
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Attachments: kafka-reconnect-time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-268) Add another reconnection condition to
the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-268:
--------------------------
Attachment: kafka-reconnect-time.patch
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Attachments: kafka-reconnect-time.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210501#comment-13210501 ]
Yang Ye commented on KAFKA-268:
-------------------------------
Sure, what about doing it like this, I added on field in the SyncProducer object, var randomGenerator = new Random()
then change the initilaization of lastConnectionTime to:
private var lastConnectionTime = System.currentTimeMillis - config.reconnectInterval + SyncProducer.randomGenerator.nextDouble() * config.reconnectInterval
Otherwise if we write
private var lastConnectionTime = System.currentTimeMillis - config.reconnectInterval + new Random().nextDouble() * config.reconnectInterval
or
private var lastConnectionTime = System.currentTimeMillis - config.reconnectInterval + new Random(System.currentTimeMillis).nextDouble() * config.reconnectInterval
we may worry about the determinism of pseudo random generator
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, random_last_connection_time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206955#comment-13206955 ]
Jun Rao commented on KAFKA-268:
-------------------------------
Some comments:
1. SyncProducerConfigShared: change reconnectTimeInterval to reconnectTimeIntervalMs and reconnect.timeInterval to reconnect.time.interval.ms, to be consistent with existing naming convention.
2. There is no real change to ByteBufferMessageSet and Utils. Please revert them.
3. The default value of reconnectTimeInterval is too small. Let's set it to 10 minutes. Also, we probably need a way to turn off time-based reconnect. How about using a negative value to indicate that? Let's also add a comment in the code to document that.
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Attachments: kafka-reconnect-time.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Jun Rao (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-268.
---------------------------
Resolution: Fixed
Fix Version/s: 0.7.1
Thanks for the patch, Yang. Just committed this.
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-268) Add another reconnection condition to
the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-268:
--------------------------
Description: Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it. (was: Add another reconnect threshold to the syncProducer: the elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it. )
Summary: Add another reconnection condition to the syncProducer: the time elapsed since last connection (was: Add )
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207281#comment-13207281 ]
Yang Ye commented on KAFKA-268:
-------------------------------
got them, nice suggestions
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Attachments: kafka-reconnect-time.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-268) Add another reconnection condition
to the syncProducer: the time elapsed since last connection
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212169#comment-13212169 ]
Jun Rao commented on KAFKA-268:
-------------------------------
Committed the improvement to trunk.
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, random_last_connection_time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-268) Add another reconnection condition to
the syncProducer: the time elapsed since last connection
Posted by "Yang Ye (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-268:
--------------------------
Attachment: random_last_connection_time.patch
> Add another reconnection condition to the syncProducer: the time elapsed since last connection
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-268
> URL: https://issues.apache.org/jira/browse/KAFKA-268
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7
> Reporter: Yang Ye
> Priority: Minor
> Fix For: 0.7.1
>
> Attachments: kafka-reconnect-time.patch, random_last_connection_time.patch, time_based_reconnect_feature.patch
>
>
> Add another reconnection condition to the syncProducer: the time elapsed since last connection. If it's larger than the pre-specified threshold, close the connection and reopen it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira