You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2012/09/01 03:27:07 UTC
[jira] [Created] (KAFKA-495) Handle topic names with "/" on Kafka
server
Neha Narkhede created KAFKA-495:
-----------------------------------
Summary: Handle topic names with "/" on Kafka server
Key: KAFKA-495
URL: https://issues.apache.org/jira/browse/KAFKA-495
Project: Kafka
Issue Type: Bug
Reporter: Neha Narkhede
If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-495:
--------------------------------
Affects Version/s: 0.8
0.7
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464349#comment-13464349 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Thanks for the 0.8 v4 patch. +1 Committed to 0.8.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch, kafka-v0.8-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-v0.8-v4.patch
Uploading kafka-v0.8-v4.patch.
This patch allows only ASCII alphanumeric chars, underscore and dash in the topic names. Also since we are arguing that the hard constraints set by us are not going to be challenged in likely usages, I have also hardcoded topic name length to have a max value of 255 since it's very difficult to exceed this length as well.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch, kafka-v0.8-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453157#comment-13453157 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Thanks for 0.8 patch v2. Some additional comments:
1. KafkaApis: Instead of adding maxTopicNameLength in the constructor, we can get that value from replicaManager.config.
2. CreateTopicCommand.createTopic(): Could we put topicNameValidator as the last parameter that defaults to an instance of topicNameValidator with the default max length. This way, most existing unit tests don't have to create a separate topicNameValidator.
3. TopicTest: Is there a particular reason that the following loop is not from 1 to 5 instead?
for (i <- 3 to 8)
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v2.patch
1. Actually the zookeeper code does not implement all the checks that I mentioned earlier. So I have implemented:
* unix constraints (don't allow / and \0 chars, and don't allow . and .. filenames)
* zookeeper constraints (dont allow chars in range \u0001-\u001F, \u007F-\u009F, \uD800-\uF8FF, \uFFF0-\uFFFF adnd \0)
2. The list of illegal chars is not configurable.
3. Regex used to check for illegal chars, . and .. filenames. It will be good if someone could take a look at the regular expression.
4. Topic name length is configurable.
5. Created a class TopicNameValidator in Utils.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448152#comment-13448152 ]
Jay Kreps commented on KAFKA-495:
---------------------------------
It would be good to first work out the regular expression for valid topic names. It might also be nice to have a helper class TopicName that we use when the string being passed is meant as a topic name which applies this regex so we can be sure we are always checking validity.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-495) Handle topic names with "/" on
Kafka server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448150#comment-13448150 ]
Swapnil Ghike edited comment on KAFKA-495 at 9/5/12 12:14 PM:
--------------------------------------------------------------
Some relevant discussion is at KAFKA-196.
was (Author: swapnilghike):
Some relevant discussion is at KAFKA-195.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v0.8-v3.patch
1. Done.
2. Thanks, should have thought of that. :\
3. Oh, that was because topicName.length == pow(2, i). The default max length is 255, and this loop would have created a name of length 256. Changed it to 1 to 6 since the former does not seem to be obvious in first glance.
4. Slight changes in two unit tests because of ReplicaManager EasyMock.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448867#comment-13448867 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Does it make sense to make the regex configurable that defaults to what Swapnil suggests?
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449297#comment-13449297 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Thanks for patch v2. Looks good overall. A couple of minor comments:
20. LogManagerTest: Instead of duplicating code, could we create a list of invalid topics and run through logManager.getOrCreate?
21. TopicNameValidator: Could we name the scala file TopicNameValidator.scala? Also, the message in InvalidTopicException should probably be something like "topic is not allowed".
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448869#comment-13448869 ]
Jay Kreps commented on KAFKA-495:
---------------------------------
I don't think it does. It is like a filesystem--we should be able to document what works and what doesn't. I don't see a lot of value to users to making it configurable, and even if you try to configure '/' to work it won't. I think it would be better to say in our documentation "the following unicode characters are valid in a topic name" and just have that be definitive...
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448166#comment-13448166 ]
Jay Kreps commented on KAFKA-495:
---------------------------------
A couple suggestions:
1. This is a fairly conservative character list. Is there a rationale for being more conservative?
2. I don't think this list should be configurable, we should be able to put in our documentation what is legal and have that always be true and not worry about what happens if someone changes the definition of legal characters.
3. One corner case we need to consider is the interaction between regular expressions and topic names.
4. This patch does ~30 passes over the topic name, it would probably be faster to use a pre-compiled regular expression
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-495) Handle topic names with "/" on
Kafka server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448374#comment-13448374 ]
Swapnil Ghike edited comment on KAFKA-495 at 9/5/12 12:21 PM:
--------------------------------------------------------------
So I am planning to insert the following regex check.
Unix constaints -
1. Ban / and \0 (both not allowed in unix, / is used as a path name component separator and \0 terminates file names in unix.)
Zookeeper constaints:-
1. The null character (\u0000) cannot be part of a path name. (This causes problems with the C binding.)
2. The following characters can't be used because they don't display well, or render in confusing ways: \u0001 - \u0019 and \u007F - \u009F.
3. The following characters are not allowed: \ud800 -uF8FFF, \uFFF0-uFFFF, \uXFFFE - \uXFFFF (where X is a digit 1 - E), \uF0000 - \uFFFFF.
4. . and .. Are not allowed as filenames. Zookeeper supports only absolute filepaths.
5. The token "zookeeper" is reserved.
In addition, I suggest the following two -
1. Ban leading and trailing whitespaces (because they confuse GUI users) - also takes care forbidding filenames composed of only whitespaces.
2. Ban leading . And .. Because unix hides those files.
There are other cases where unfortunate errors may happen in ways not related to kafka or zookeeper, and programmers would need to take care about them.
was (Author: swapnilghike):
So I am planning to insert the following regex check.
Unix constaints -
1. Ban / (both not allowed in unix, / is used as a path name component separator and \0 terminates file names in unix.)
Zookeeper constaints:-
1. The null character (\u0000) cannot be part of a path name. (This causes problems with the C binding.)
2. The following characters can't be used because they don't display well, or render in confusing ways: \u0001 - \u0019 and \u007F - \u009F.
3. The following characters are not allowed: \ud800 -uF8FFF, \uFFF0-uFFFF, \uXFFFE - \uXFFFF (where X is a digit 1 - E), \uF0000 - \uFFFFF.
4. . and .. Are not allowed as filenames. Zookeeper supports only absolute filepaths.
5. The token "zookeeper" is reserved.
In addition, I suggest the following two -
1. Ban leading and trailing whitespaces (because they confuse GUI users) - also takes care forbidding filenames composed of only whitespaces.
2. Ban leading . And .. Because unix hides those files.
There are other cases where unfortunate errors may happen in ways not related to kafka or zookeeper, and programmers would need to take care about them.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike reassigned KAFKA-495:
-----------------------------------
Assignee: Swapnil Ghike
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v0.8-v2.patch
- After discussing with Jun, inserted topic validation in createTopic() instead of LogManager in 0.8.
- Created a singleton Topic in Utils/Topic.scala, topicNameValidators in KafkaConfig and CreateTopicCommand default to the maxNameLen in this singleton. The value at both these locations can be overridden.
- The check in LogManager.createLog is not necessary anymore because the topic would be validated by then, removed this check.
- Added a new unit test in test/utils for topic validation.
- Tested this change with bin/kafka-create-topic.sh for fun.
- Opened KAKFA-505 to deal with the TopicMetaDataResponse errorcode.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v1.patch
Created two server config properties -
1. Black list of characters that are not allowed in the topic names. Currently it excludes alphanumeric characters, _, -, (, ), [, ], { and }. Whitespaces are blacklisted.
2. Maximum length of a topic name.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450315#comment-13450315 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Thanks for the patch. Committed to trunk. Could you port it to 0.8 too?
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (KAFKA-495) Handle topic names with "/" on
Kafka server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on KAFKA-495 started by Swapnil Ghike.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448374#comment-13448374 ]
Swapnil Ghike commented on KAFKA-495:
-------------------------------------
So I am planning to insert the following regex check.
Unix constaints -
1. Ban / (both not allowed in unix, / is used as a path name component separator and \0 terminates file names in unix.)
Zookeeper constaints:-
1. The null character (\u0000) cannot be part of a path name. (This causes problems with the C binding.)
2. The following characters can't be used because they don't display well, or render in confusing ways: \u0001 - \u0019 and \u007F - \u009F.
3. The following characters are not allowed: \ud800 -uF8FFF, \uFFF0-uFFFF, \uXFFFE - \uXFFFF (where X is a digit 1 - E), \uF0000 - \uFFFFF.
4. . and .. Are not allowed as filenames. Zookeeper supports only absolute filepaths.
5. The token "zookeeper" is reserved.
In addition, I suggest the following two -
1. Ban leading and trailing whitespaces (because they confuse GUI users) - also takes care forbidding filenames composed of only whitespaces.
2. Ban leading . And .. Because unix hides those files.
There are other cases where unfortunate errors may happen in ways not related to kafka or zookeeper, and programmers would need to take care about them.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449993#comment-13449993 ]
Jay Kreps commented on KAFKA-495:
---------------------------------
+1
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v3.patch
Thanks, made the changes.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448858#comment-13448858 ]
Jay Kreps commented on KAFKA-495:
---------------------------------
That sounds reasonable. I would actually argue for allowing anything unix+zk will allow. I agree that leading or trailing whitespace can be confusing and dots are odd, but I think forbidding or allowing that should be done at a higher level than kafka, we should allow anything we can tolerate. LiKafka wrapper can support a more conservative set of restrictions and naming conventions based on what we think names should look like.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-495:
--------------------------
Fix Version/s: (was: 0.7.1)
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.2
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch, kafka-v0.8-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Fix Version/s: 0.7.1
0.8
Labels: bugs (was: )
Status: Patch Available (was: In Progress)
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-495:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for patch v3 for 0.8. Committed to 0.8.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v4.patch
Changed the property string in KafkaConfig for maxTopicNameLength.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463259#comment-13463259 ]
Swapnil Ghike commented on KAFKA-495:
-------------------------------------
Reopening this issue because of additional constraints imposed due to jmx mbean Object naming spec and our own highwatermark handling.
JMX Mbean naming spec is tedious to implement. The full rules are at http://docs.oracle.com/javase/7/docs/api/javax/management/ObjectName.html. The easy way to make sure that we don't break any rule is to ban the characters colon (:), asterisk (*), question mark (?), comma (,), equals (=), quote ("), backslash (\), newline ('\n'). There does not seem to be any restrictions on single quote ('), period (.) and the rest.
For Highwatermark handling, we need to ban whitespaces [ \t\r\n\f].
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452074#comment-13452074 ]
Jun Rao commented on KAFKA-495:
-------------------------------
Thanks for the patch for 0.8. Overall, it looks good. Some minor comments.
80. LogManager:
80.1. We don't need to verify the topic name in getLog() since logs are not created there. Verifying it in createLog is enough.
80.2. Get rid of unused imports.
81. TopicNameValidator: get rid of "/**/"
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449119#comment-13449119 ]
Joel Koshy commented on KAFKA-495:
----------------------------------
I agree with not having a configurable topic name spec. I think that would be applicable if we decide to have an extremely restrictive spec by default (e.g., only alpha-numeric) in which case many users would want to override that. However the spec that Swapnil has sounds reasonably broad.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448150#comment-13448150 ]
Swapnil Ghike commented on KAFKA-495:
-------------------------------------
Some relevant discussion is at KAFKA-195.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v1.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnil Ghike updated KAFKA-495:
--------------------------------
Attachment: kafka-495-v0.8.patch
Attached the 0.8 patch. It was taken with git --no-prefix, hope it works.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Swapnil Ghike (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463303#comment-13463303 ]
Swapnil Ghike commented on KAFKA-495:
-------------------------------------
After a bit of discussion, decided to ALLOW only ASCII alphanumeric characters and underscore and dash.
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-495:
--------------------------
Fix Version/s: 0.7.2
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.2
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch, kafka-v0.8-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-495) Handle topic names with "/" on Kafka
server
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-495.
-------------------------
> Handle topic names with "/" on Kafka server
> -------------------------------------------
>
> Key: KAFKA-495
> URL: https://issues.apache.org/jira/browse/KAFKA-495
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7, 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Labels: bugs
> Fix For: 0.8, 0.7.1
>
> Attachments: kafka-495-v0.8.patch, kafka-495-v0.8-v2.patch, kafka-495-v0.8-v3.patch, kafka-495-v1.patch, kafka-495-v2.patch, kafka-495-v3.patch, kafka-495-v4.patch
>
>
> If a producer publishes data to topic "foo/foo", the Kafka server ends up creating an invalid directory structure on the server. This corrupts the zookeeper data structure for the topic - /brokers/topics/foo/foo. This leads to rebalancing failures on the consumer as well as errors on the zookeeper based producer.
> We need to harden the invalid topic handling on the Kafka server side to avoid this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira