You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (JIRA)" <ji...@apache.org> on 2012/10/12 02:31:02 UTC

[jira] [Created] (KAFKA-569) Cleanup kafka.utils.Utils

Jay Kreps created KAFKA-569:
-------------------------------

             Summary: Cleanup kafka.utils.Utils
                 Key: KAFKA-569
                 URL: https://issues.apache.org/jira/browse/KAFKA-569
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.8
            Reporter: Jay Kreps
            Assignee: Jay Kreps


kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475430#comment-13475430 ] 

Neha Narkhede commented on KAFKA-569:
-------------------------------------

Looks good, just one thing that I missed in the first review - I see you added a counter to DefaultEventHandler for the round-robin load-balancing. For a long running producer, the counter could overflow to a negative value. I think we need to reset it to 0, no ?

+1 otherwise.
                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede closed KAFKA-569.
-------------------------------

    
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-569:
----------------------------

    Attachment: KAFKA-569-v2.patch

Neha, thanks for the quick review!

New patch:
- Added the comment you requested
- Rebased
- Changed naming of ISR to be consistent (we had, for example, LeaderAndIsrRequest but LeaderAndISRResponse)
                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475443#comment-13475443 ] 

Jay Kreps commented on KAFKA-569:
---------------------------------

Oh my, good catch.

BTW, in fixing this I found a rare bug in DefaultPartitioner which does a similar thing. The cause is that we take
  math.abs(key.hashCode) % numPartitions
The problem with this is that since the creators of java hate all living things they made java.lang.Math.abs return a negative number when given Integer.MIN_VALUE (since it has no equivalent positive integer). Sooner or later hashCode() will return a negative value, which would cause this.

I am adding a new method in Utils (how apt)
 def abs(n: Int) = n & 0x7fffffff

That should fix this. Not sure if we ever saw this happen in practice.
                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475445#comment-13475445 ] 

Neha Narkhede commented on KAFKA-569:
-------------------------------------

>> The problem with this is that since the creators of java hate all living things they made java.lang.Math.abs return a negative number when given Integer.MIN_VALUE (since it has no equivalent positive integer). Sooner or later hashCode() will return a negative value, which would cause this.

:O (Stumped and don't know what else to say). Great catch !
                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-569:
----------------------------

    Attachment: KAFKA-569.patch

This patch has a large number of trivial changes.

Here is a summary:
- Make bad names good
- Create kafka.api.ApiUtils and move all protocol parsing related methods there
- Get rid of the encoding argument in ApiUtils since our protocol requires UTF8, so making this changable is not good.
- Move zookeeper method into ZkUtils, delete random duplicate version of that code elsewhere
- Move SyncJSON which was inside Utils (!) into its own file
- Create ClientUtils and put getTopicMetadata() there
- Create CommandLineUtils and put command line stuff there
- Delete a few silly, unused methods
- Move getTopicPartition into LogManager since it is an implementation detail of LogManager
- Move getAllBrokersFromBrokerList into BrokerPartitionInfo, the only place it is used
- Get rid of Utils.getNextRandomInt. Since random is synchronized it is better to not have a single one.
- Changed DefaultEventHandler to user a counter rather than Utils.randomInt--this will be more evenly distributed since it is true round-robin.
- Delete ~5 custom property parsing methods that were jammed into Utils. Add a generic method VerifiableProperties.getMap that replaces them all.
                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474770#comment-13474770 ] 

Neha Narkhede commented on KAFKA-569:
-------------------------------------

Looks great, thanks for contributing this cleanup patch !
Just one minor suggestion - Can we add a description in ClientUtils similar to the one in Utils and ApiUtils ?

                
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (KAFKA-569) Cleanup kafka.utils.Utils

Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede resolved KAFKA-569.
---------------------------------

    Resolution: Fixed
    
> Cleanup kafka.utils.Utils
> -------------------------
>
>                 Key: KAFKA-569
>                 URL: https://issues.apache.org/jira/browse/KAFKA-569
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira