[jira] [Updated] (KAFKA-554) Move all per-topic configuration into ZK and add to the CreateTopicCommand

["Jay Kreps (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-554:
----------------------------

    Attachment: KAFKA-554-v2.patch

New patch v2. Addresses Jun's comments and rebases.

1. TopicCommand: Made it a bit more clear which options apply to which actions.

2. AdminUtils:
2.1 Fixed typo
2.2 Moved serialization to JSON. The APIs still take Properties instances as that is a higherarchical map. However the data stored in ZK is now JSON. One question is what the format of the change notifications should be. I added a proposal to the kafka zk wiki, please take a look.

3. TopicConfigManager:
3.1 add the missing > in the following comment.
 * /brokers/topics/<topic_name/config
3.2 Yeah but this component should stand alone ideally.
3.3.1 Done
3.3.2 Good call, added that optimization.
3.3.3 I wasn't sure if deleting all znodes would lead to the sequence starting over. I am told it won't so removed this.
3.3.4 "It seems that sequential nodes under /brokers/config_changes are only deleted when there is a new config change. So, they are not always deleted after the configured expiration time." Yes. My rational is that I want to clean up just to prevent notifications from piling up forever. But if no new notifications come in then they can't keep piling up. I could also do this by scheduling the deletion but the problem with that is knowing whether you have already scheduled it (you don't want to schedule it over and over). I could also add a background thread dedicated to cleanup, but it doesn't seem worth it for the added complexity.
3.4. Nice call, sorted it
4. Hmm, that doesn't actually compile for me. 
5. Not sure that helps. Config is a contract so we can't change these values in the future so just having the value is more straight-forward (one fewer layers of indirection)
6. Removed ModifyTopicKey
7. ZkUtils: removed imports
8. Yeah, that slipped in. Removed.
9. ReplicaFetchTest.logsMatch(): renamed tandp it to topicAndPart?
                
> Move all per-topic configuration into ZK and add to the CreateTopicCommand
> --------------------------------------------------------------------------
>
>                 Key: KAFKA-554
>                 URL: https://issues.apache.org/jira/browse/KAFKA-554
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>              Labels: project
>             Fix For: 0.8.1
>
>         Attachments: KAFKA-554-v1.patch, KAFKA-554-v2.patch
>
>
> We have a number of per-topic configurations that control message retention and flush interval. Here is the list of properties I find in KafkaConfig that appear to be per-topic:
>   topic.log.file.size
>   topic.log.roll.hours
>   topic.log.retention.hours
>   topic.log.retention.size
>   topic.flush.intervals.ms
> Currently we specify these in server.properties. This is not a good solution as it requires a rolling bounce of the cluster to make a change, which just doesn't scale to having hundreds of topics. Also the map encoded in a CSV string is kind of hacky.
> We should move these into ZK in some kind of JSON blob that allows easily adding new per-topic configs and we should remove these from server.properties.
> It would be good to start with a wiki design and get consensus on that first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira