You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2011/08/15 05:44:27 UTC

[jira] [Created] (KAFKA-101) Avoid creating a new topic by the consumer

Avoid creating a new topic by the consumer
------------------------------------------

                 Key: KAFKA-101
                 URL: https://issues.apache.org/jira/browse/KAFKA-101
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Jun Rao


Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153952#comment-13153952 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

I realized on Friday that I had neglected to fix getOffsets.  So calling getOffsets on a non-existent topic creates a file too.  I wrote the fix yesterday - I will see if I can merge the two together and post an updated patch with tests.  

                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101_v2.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jun Rao updated KAFKA-101:
--------------------------

       Resolution: Fixed
    Fix Version/s: 0.7
           Status: Resolved  (was: Patch Available)

Thanks, Taylor. Just committed this.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>             Fix For: 0.7
>
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156243#comment-13156243 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

So - I rolled out my version of this patch which was applied to 0.6.  In dev and stage, it worked just great.   In production, we it appears we ended up with a very mysterious occurrence.  

Topics appeared to be created and the first message delivered was corrupted.  Instead of having the usual binary data at the start of a topic file, the beginning of the file held the partial contents of the first message.

I'm going to start looking into how that might even be possible - I thought to post here in case in other stages of development maybe you saw this problem and/or fixed it which might help me pinpoint which part of the code could be responsible that I mistakenly broken.

To be clear, if the first message delivered was let's say "One fine day in the middle of the night" then the contents of our topic files appeared as so:

-----------------------------------
ay in the middle of the night
------------------------------------

When it should have been something like:
---------------------------------
[binary stuff]One fine day in the middle of the night
------------------------------------
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>             Fix For: 0.7
>
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149435#comment-13149435 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

sorry, I am not seeing a SimpleConsumer test.  Do you have a specific file in mind?  Maybe log/LogOffsetTest.scala?
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150246#comment-13150246 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

Actually another thing. If a topic doesn't already exist, the consumer shouldn't create the ZK path /broker/topics/topic. Currently, we do the following in ZookeeperConsumerConnector.consume

      ZkUtils.makeSurePersistentPathExists(zkClient, partitionPath)

We should remove that line. There are a couple of things to check.
(1) If the consumer can still get the watcher triggered when the topic is created (by a producer). I think this should happen since ZKClient registers an Exists watcher if path doesn't exist. But we should check.
(2) There are probably places in the rebalance code where we should treat missing the ZK path  /broker/topics/topic properly.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149424#comment-13149424 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

I added a test for the empty log in LogManager.  You want a similar test in SimpleConsumer also?
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jun Rao updated KAFKA-101:
--------------------------

    Attachment: KAFKA-101_v3.patch

Merged the v2 patch and the getoffset patch into a v3 patch.
1. get rid of unused method LogManager.getLogOrGenerateFake
2. If the topic hasn't been created, probably only getting the latest or the earliest offset make sense. Otherwise, we should return an empty array.
3. Optimized the import on several files.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156383#comment-13156383 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

Unfortunately, I cannot.  I have gone ahead and updated our clients to speak 0.7 properly so we are going to move soon.  However it's going to be hard to justify to roll out just 0.7 with this patch without any changes "just to see" if the problem resurfaces.  We might go ahead and do it on one machine.  Anyway that will have to wait until at best sometime next week since we will have to rollout updated clients that can speak 0.6 and 0.7 first.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>             Fix For: 0.7
>
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jun Rao updated KAFKA-101:
--------------------------

    Attachment: KAFKA-101_v2.patch

Attache patch v2. Took Taylor's patch and added (1) a SimpleConsumer level test in PrimitiveApiTest (2) don't let consumer create the topic path under /brokers/topics (only producer can create it).
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101_v2.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Taylor Gautier updated KAFKA-101:
---------------------------------

               Labels: patch  (was: )
    Affects Version/s: 0.7
               Status: Patch Available  (was: Open)

Fixes KAFKA-101 but the core tests for ZK fail - I think because other tests are expecting a getOrCreate call to connect to ZK and it no longer does that.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Taylor Gautier updated KAFKA-101:
---------------------------------

    Attachment: KAFKA-101-getoffsets.patch
    
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154043#comment-13154043 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

lgtm - I tried to make Log.getLog(...) more scala like and have it return Option[Log] but that somehow caused other tests to fail, my scala fu is not yet strong.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149442#comment-13149442 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

We can add it in PrimitiveApiTest.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153957#comment-13153957 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

Ok - your commit looks good.  I've added the get offsets patch which should be applied after yours.
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Taylor Gautier updated KAFKA-101:
---------------------------------

    Attachment: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
    
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149426#comment-13149426 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

Yes, since a test in SimpleConsumer will further test the wire protocol. Thanks,
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Taylor Gautier (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154797#comment-13154797 ] 

Taylor Gautier commented on KAFKA-101:
--------------------------------------

great! :)
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>             Fix For: 0.7
>
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149414#comment-13149414 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

Thanks for the patch.

The test failures seem to be transient and are the results of time dependencies, which we should fix separately.

Could you add a unit test on SimpleConsumer to consume a non-existing topic/partition? It should return an empty messageSet. Other than that, the patch looks good. 
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156377#comment-13156377 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

Taylor, this would suggest a synchronization issue during append to the log. Looking at the code, the append is always synchronized on the lock of the segment file. Could you reproduce the problem on 0.7? 
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>             Fix For: 0.7
>
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101-getoffsets.patch, KAFKA-101_v2.patch, KAFKA-101_v3.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (KAFKA-101) Avoid creating a new topic by the consumer

Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153917#comment-13153917 ] 

Jun Rao commented on KAFKA-101:
-------------------------------

Taylor, could you review patch v2? Thanks,
                
> Avoid creating a new topic by the consumer
> ------------------------------------------
>
>                 Key: KAFKA-101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-101
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>              Labels: patch
>         Attachments: 0002-KAFKA-101-Avoid-creating-a-new-topic-by-the-consumer.patch.0.7, KAFKA-101_v2.patch
>
>
> Currently, if  a consumer consumes a topic and the topic doesn't exist, the topic is created automatically. Sometimes this can be confusing. Often, an ad hoc user may put a wrong topic name in the consumer and thus create many unnecessary topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira