You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/08/13 19:08:38 UTC
[jira] [Updated] (KAFKA-351) Refactor some new components
introduced for replication
[ https://issues.apache.org/jira/browse/KAFKA-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-351:
--------------------------
Attachment: kafka-351_v1.patch
Attach patch v1. An overview of the patch.
A. Use synchronized instead of lock for synchronization. The latter has more functionality, but more overhead. See
http://blog.rapleaf.com/dev/2011/06/16/java-performance-synchronized-vs-lock/
B. Partition: Consolidated all reads/writes to leader/ISR in Partition and all accessed are synchronized. This makes sure that leader/ISR values don't change while doing the following operations.
- makeLeader
- makerFollower
- maybeShrinkISR
- updateLeaderHWAndMaybeExpandISR (for maintaining remote replicas)
- checkEnoughReplicasReachAnOffset (for checking if a produce request is satisfied)
This means that Partition has to access ReplicaManager. In some sense, Partition probably should be a nested class under ReplicaManager since it needs to access several members of ReplicaManager. However, this will make ReplicaManager too big.
C. RepicaManager:
- Moved most per partition operations to Partition.
- Cleaned up public methods and added a few helper methods for getting partition and replica.
- Use ConcurrentHashMap for managing all partitions.
D. KafkaApis: Removed callbacks in the constructer. Instead, call methods in ReplicaManager directly.
E. Replica:
- Changed to the new getter/setter style for logEndOffset and highWatermark.
- Local replica doesn't need to set logEndOffset anymore since the logEndOffsetUpdateTime for local replica is never used.
F. BrokerPartitionInfo: Partition is now a complex structure and is supposed to be used only on the server side. Create a simpler PartitionAndLeader class for client usage.
G. KafkaZookeeper: Removed ensurePartitionLeaderOnThisBroker(). The checking of the existence of a leader is now done by replicaManager.leaderReplicaIfLocalOrException(), which is cheaper since it doesn't access ZK.
H. ISRExpirationTest: Remove the test testISRExpirationForMultiplePartitions. It doesn't seem to add additional value since ISR expiration is always done on a per partition basis.
I. SyncProducerTest: Remove the test testProducRequestForUnknowTopic since the logic is always covered by #1 in testProduceCorrectlyReceivesResponse.
J. TestUtils: Added a helper method leaderLocalOnBroker() that more reliably ensures that a leader exists on a broker.
K. TopicCounTest: removed since it's not doing any useful test.
L. ZookeeperConsumerConnectorTest.testCompressionSetConsumption(): removed the part that tests consumer timeout since it's covered in testBasic already.
> Refactor some new components introduced for replication
> --------------------------------------------------------
>
> Key: KAFKA-351
> URL: https://issues.apache.org/jira/browse/KAFKA-351
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Jun Rao
> Attachments: kafka-351_v1.patch
>
>
> Jay had some good refactoring suggestions as part of the review for KAFKA-46. I'd like to file this umbrella JIRA with individual sub tasks to cover those suggestions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira