You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Joel Koshy (JIRA)" <ji...@apache.org> on 2011/08/15 21:13:35 UTC
[jira] [Created] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Make whitelist/blacklist mirror configs more consistent
-------------------------------------------------------
Key: KAFKA-103
URL: https://issues.apache.org/jira/browse/KAFKA-103
Project: Kafka
Issue Type: Improvement
Affects Versions: 0.7
Reporter: Joel Koshy
Priority: Minor
The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jun Rao (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-103.
---------------------------
Resolution: Fixed
Fix Version/s: 0.7
I think the concerns raised here can be addressed with clear documentation on how mirroring is done.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Assignee: Jun Rao
> Priority: Minor
> Fix For: 0.7
>
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-103:
-----------------------------
Attachment: kafka-103-patch.v1
Sorry about deleting that.. I forgot to update the property in system test.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-103:
-----------------------------
Attachment: kafka-103-patch.v1
Hmm.. looks like the mirror topics patch actually ignored the numthreads in the whitelist. In any event, this patch allows a global numthreads for the embedded consumer.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088145#comment-13088145 ]
Jun Rao commented on KAFKA-103:
-------------------------------
However, we haven't officially released anything that exposes whitelist in the config. We are just iterating in trunk.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-103:
-----------------------------
Attachment: kafka-103-incremental-v2.patch
Note that this is an incremental patch.
That is a good point - i.e., we have told clients that we are changing the old whitelist config to mirror.topics.whitelist. If they simply do a rename then nothing is going to get mirrored, so I agree that backward-compatibility is important.
btw, I think a better fix (that would also help KAFKA-104) is to disallow invalid topics, but it would be better to think about that more carefully.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086682#comment-13086682 ]
Jun Rao commented on KAFKA-103:
-------------------------------
Thanks, Joel. Just committed this.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087299#comment-13087299 ]
Neha Narkhede commented on KAFKA-103:
-------------------------------------
The behavior of the mirror.topics.whitelist is backwards incompatible. If some users currently have a whitelist setup with the old format (topic1:numPartitions1, topic2:numPartitions2), it is ideal to just rename "embedded.consumer.topics" to "mirror.topics.whitelist". Instead, this change forces those users to change their white-listed topics list to strip off the number of partitions. It would be ideal if the number of partitions is just ignored, with a little warning message. That way the change is backwards compatible.
If a user just renames "embedded.consumer.topics" to "mirror.topics.whitelist" and leaves the number of partitions, currently the corp replica behavior is to not mirror anything at all. That is unintuitive.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088207#comment-13088207 ]
Jun Rao commented on KAFKA-103:
-------------------------------
In summary, I think it's simpler if we enforce a single format for any property value. In this case, to help users identify problems, we can probably log the whitelisted and blacklisted topics, if any.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088032#comment-13088032 ]
Joel Koshy commented on KAFKA-103:
----------------------------------
At least per kafka-103-patch.v1 it is not totally ignored. So if the whitelist config contains: SomeTopic:1 then SomeTopic will not get mirrored because it will fail the isTopicAllowed filter in KafkaServerStartable. i.e., no topics will get mirrored.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088144#comment-13088144 ]
Neha Narkhede commented on KAFKA-103:
-------------------------------------
Jun, even if we don't want to support the older format, the behavior of silently not mirroring any data is also not acceptable. What would be helpful is to either log it as a warning or throw an InvalidConfigException.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jay Kreps (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps reassigned KAFKA-103:
-------------------------------
Assignee: Jun Rao
Is this complete?
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Assignee: Jun Rao
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-103:
-----------------------------
Attachment: (was: kafka-103-patch.v1)
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-103) Make whitelist/blacklist mirror
configs more consistent
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088016#comment-13088016 ]
Jun Rao commented on KAFKA-103:
-------------------------------
I actually don't think that we need to make this backward compatible because the # of threads specified in the old format will be ignored in the new config. If we support the old format, users will assume that the number of consumer threads is in effect. It's better to make this incompatible so the users can realize there is a problem.
> Make whitelist/blacklist mirror configs more consistent
> -------------------------------------------------------
>
> Key: KAFKA-103
> URL: https://issues.apache.org/jira/browse/KAFKA-103
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Joel Koshy
> Priority: Minor
> Attachments: kafka-103-incremental-v2.patch, kafka-103-patch.v1
>
>
> The blacklist config for kafka mirrors is a comma separated list of topics. However, the whitelist config is a comma-separated list of "topics:numthreads" pairs, which allows for a multi-threaded consumer in the mirror. It will be good to keep the two configs consistent in format. So, we can make the whitelist config a comma-separated list of topics and provide a config (say, kafka.mirror.consumer.threads) that will specify the number of threads to use for all topics in the whitelist (if present).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira