You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (JIRA)" <ji...@apache.org> on 2012/10/12 02:31:02 UTC
[jira] [Created] (KAFKA-569) Cleanup kafka.utils.Utils
Jay Kreps created KAFKA-569:
-------------------------------
Summary: Cleanup kafka.utils.Utils
Key: KAFKA-569
URL: https://issues.apache.org/jira/browse/KAFKA-569
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8
Reporter: Jay Kreps
Assignee: Jay Kreps
kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475430#comment-13475430 ]
Neha Narkhede commented on KAFKA-569:
-------------------------------------
Looks good, just one thing that I missed in the first review - I see you added a counter to DefaultEventHandler for the round-robin load-balancing. For a long running producer, the counter could overflow to a negative value. I think we need to reset it to 0, no ?
+1 otherwise.
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede closed KAFKA-569.
-------------------------------
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-569:
----------------------------
Attachment: KAFKA-569-v2.patch
Neha, thanks for the quick review!
New patch:
- Added the comment you requested
- Rebased
- Changed naming of ISR to be consistent (we had, for example, LeaderAndIsrRequest but LeaderAndISRResponse)
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475443#comment-13475443 ]
Jay Kreps commented on KAFKA-569:
---------------------------------
Oh my, good catch.
BTW, in fixing this I found a rare bug in DefaultPartitioner which does a similar thing. The cause is that we take
math.abs(key.hashCode) % numPartitions
The problem with this is that since the creators of java hate all living things they made java.lang.Math.abs return a negative number when given Integer.MIN_VALUE (since it has no equivalent positive integer). Sooner or later hashCode() will return a negative value, which would cause this.
I am adding a new method in Utils (how apt)
def abs(n: Int) = n & 0x7fffffff
That should fix this. Not sure if we ever saw this happen in practice.
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475445#comment-13475445 ]
Neha Narkhede commented on KAFKA-569:
-------------------------------------
>> The problem with this is that since the creators of java hate all living things they made java.lang.Math.abs return a negative number when given Integer.MIN_VALUE (since it has no equivalent positive integer). Sooner or later hashCode() will return a negative value, which would cause this.
:O (Stumped and don't know what else to say). Great catch !
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-569:
----------------------------
Attachment: KAFKA-569.patch
This patch has a large number of trivial changes.
Here is a summary:
- Make bad names good
- Create kafka.api.ApiUtils and move all protocol parsing related methods there
- Get rid of the encoding argument in ApiUtils since our protocol requires UTF8, so making this changable is not good.
- Move zookeeper method into ZkUtils, delete random duplicate version of that code elsewhere
- Move SyncJSON which was inside Utils (!) into its own file
- Create ClientUtils and put getTopicMetadata() there
- Create CommandLineUtils and put command line stuff there
- Delete a few silly, unused methods
- Move getTopicPartition into LogManager since it is an implementation detail of LogManager
- Move getAllBrokersFromBrokerList into BrokerPartitionInfo, the only place it is used
- Get rid of Utils.getNextRandomInt. Since random is synchronized it is better to not have a single one.
- Changed DefaultEventHandler to user a counter rather than Utils.randomInt--this will be more evenly distributed since it is true round-robin.
- Delete ~5 custom property parsing methods that were jammed into Utils. Add a generic method VerifiableProperties.getMap that replaces them all.
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474770#comment-13474770 ]
Neha Narkhede commented on KAFKA-569:
-------------------------------------
Looks great, thanks for contributing this cleanup patch !
Just one minor suggestion - Can we add a description in ClientUtils similar to the one in Utils and ApiUtils ?
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-569) Cleanup kafka.utils.Utils
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede resolved KAFKA-569.
---------------------------------
Resolution: Fixed
> Cleanup kafka.utils.Utils
> -------------------------
>
> Key: KAFKA-569
> URL: https://issues.apache.org/jira/browse/KAFKA-569
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-569.patch, KAFKA-569-v2.patch
>
>
> kafka.utils.Utils is a real mess. It is full of odd little pieces of business logic dropped there. We should clean this up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira