You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2012/05/26 04:23:23 UTC
[jira] [Created] (KAFKA-350) Improve the high water mark
maintenance to store high watermarks for all partitions in a single file on
disk
Neha Narkhede created KAFKA-350:
-----------------------------------
Summary: Improve the high water mark maintenance to store high watermarks for all partitions in a single file on disk
Key: KAFKA-350
URL: https://issues.apache.org/jira/browse/KAFKA-350
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8
Reporter: Neha Narkhede
KAFKA-46 introduced per partition leader high watermarks. But it stores those in one file per partition. A more performant solution would be to store all high watermarks in a single file on disk
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch, kafka-350-v4.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v4.patch
21. Not really. updateInfo() is not the only API that throws NoBrokersForPartitionException. partitionAndCollate can encounter that exception while calling the getPartitionListForTopic() API. But you raised a good point here. What partitionAndCollate() should do (given its current design) is to log a retry warning for recoverable exceptions and return None, so that they are included in the retry. For non-recoverable exceptions, it should just throw the exception to the handle API and mark it failed.
30. Added an error message for non-positive values for request timeout. Al though a value of zero means infinite timeout
31. I guess so. Changed it back to IllegalStateException
32. Removed socket timeout related stuff from ProducerPerformance
33. Removed the import.
34. Hmm, that can happen if you try to create a topic when the brokers haven't yet registered in zookeeper. The system tests waits for 2 seconds, which should be enough for couple of brokers to startup in the normal case. I haven't been able to reproduce this. Its possible that this is something that can be fixed in the system test. File a bug if you see it.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch, kafka-350-v4.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v1.patch
This patch contains fixes to bugs in various components to make message replication work in the presence of controlled server failures. The system test under system_test/single_host_multiple_brokers passes with server failures enabled.
This patch contains the following changes -
1. Topic metadata request bug
1.1. While responding to a topic metadata request bug, the server uses the AdminUtils to query ZK for the host-port info for the leader and other replicas. However, it can happen that one of the brokers in the replica is offline and hence its broker registration in ZK is unavailable. Since the system test simulates exactly this scenario, the server's request handler thread was exiting due to a NoNodeException while reading the /brokers/ids/[id] path. The general problem seems to be handling error codes correctly in the request handlers and sending them to the client. I think once KAFKA-402 is resolved, all error codes will be handled cleanly by the server. For now, I've modified topic metadata response to have an error code per topic as well as per partition. So, if the leader is not available for some partitions, it will set the LeaderNotAvailable error code at the partition level. If some other replica is not available, it will set the ReplicaNotAvailable error code in the response instead of throwing an exception and exiting.
1.2. On the producer side, it fails the produce request retry if the leader for that partition is not available. It logs all other kinds of errors and ignores them.
1.3. Changed some unit tests in AdminTest to elect a leader before making topic metadata requests, so that the test passes.
1.4. Fixed another bug in the deserialization of the topic metadata response that read the versionId and errorCode in incorrect order.
1.5. In DefaultEventHandler, during a retry, it relied on the topic cache inside BrokerPartitionInfo for refreshing topic metadata. However, if the topic cache is empty (when no previous send request has succeeded), it doesn't end up refreshing the metadata for the required topics in the produce request. Fixed this by passing the list of topics explicitly in the call to updateInfo()
1.6. In general, it seems like a good idea to have a global error code for the entire response (to handle global errors like illegal request format), then a per topic error code (to handle error codes like UnknownTopic), a per-partition error code (to handle partition-level error codes like LeaderNotAvailable) and also a per-replica error code (to handle ReplicaNotAvailable). Jun had a good suggestion about the format of TopicMetadata response. I would like to file another JIRA to improve the format of topic metadata request.
2. Bug in TestUtils waitUntilLeaderIsElected was signalling a condition object without acquiring the corresponding lock. This error message was probably getting masked since we had turned off ERROR log4j level for unit tests. Fixed this.
3. Log recovery
3.1. Removed the highWatermark() API in Log.scala, since we used “highwatermark” to indicate the offset of the latest message flushed to disk. This was causing the server to prevent the replicas from fetching unflushed data on the leader. With replication, we use highwatermark to denote the offset of the latest committed message. I removed all references to the older API (highWatermark). To remain consistent with setHW, I added an API called getHW. As I understand it, these APIs will be refactored/renamed to match conventions, when KAFKA354 is resolved
3.2. To handle truncating a log segment, I added a truncateUpto API that takes in the checkpointed highwatermark value for that partition, computes the correct end offset for that log segment and truncates data after the computed end offset
3.3. Improved log recovery to delete segments that have start offset > highwatermark
3.4. Fixed logEndOffset to return the absolute offset of the last message in the log for that partition.
3.5. To limit the changes in this patch, it does not move the highwatermarks for all partitions to a single file. This will be done as part of KAFKA-405
3.6. Added a LogRecoveryTest to test recovery of log segments with and without failures
4. Config options
4.1. We might want to revisit the defaults for all config options. For example, the isr.keep.in.sync.time.ms defaulted to 30s which seems way too optimistic. While running the system tests, most messages timed out since the isr.keep.in.sync.time.ms and the frequency of bouncing the replicas was also roughly 30s. The socket timeout for the producer also defaults to 30s which seems very long to block a producer for.
4.2. The socket.timeout.ms should probably be set to producer.request.ack.timeout.ms, if it is a non-negative value. If producer.request.ack.timeout.ms = -1, socket.timeout.ms should probably default to a meaningful value. While running the system test, I observed that the socket timeout was 30s and the producer.request.ack.timeout.ms was -1, which means that the producer would always block for 30s if the server failed to send back an ACK. Since the frequency of bouncing the brokers was also 30s, most produce requests were timing out.
5. Socket server bugs
5.1. A bug in processNewResponses() causes the SocketServer to process responses inspite of it going through a shutdown. It probably makes sense to let outstanding requests timeout and shutdown immediately
5.2. Another bug in processNewResponses() causes it to go in an infinite loop when the selection key processing throws an exception. It failed to move to the next key in this case. I fixed it by moving the next key processing in the finally block.
6. Moved the zookeeper client connection from startup() API in KafkaZookeeper to startup() API in KafkaServer.scala. This is because the ReplicaManager is instantiated right after KafkaZookeeper and was passed in the zkclient object created by KafkaZookeeper. Since KafkaZookeeper started the zkclient only in startup() API, ReplicaManager's API's got NPE while trying to access the passed-in zkclient. With the fix, we can create the zookeeper client connection once in the KafkaServer startup, pass it around and tear it down in the shutdown API of KafkaServer.
7. System testing
7.1. Hacked ProducerPerformance to turn off forceful use of the async producer. I guess ProducerPerformance has grown over time into a complex blob of if-else statements. Will file another JIRA to refactor it and fix it so that all config options work well with both sync and async producer.
7.2. Added producer ACK config options to ProducerPerformance. Right now, they are hardcoded. I'm hoping this can be fixed with the above JIRA.
7.3. The single_host_multiple_brokers system test needs to be changed after KAFKA-353. Basically, it needs to count the successfully sent messages correctly. Right now, there is no good way to do this in the script. One way is to have the system test script grep the logs to find the successfully ACKed producer requests. To get it working for now, hacked it to use sync producer. Hence, before these issues are fixed it will be pretty slow.
7.4. Improved the system test to start the console consumer at the very end of the test for verification
8. Another thing that can be fixed in KAFKA-402 is the following -
8.1. When a broker gets a request for a partition for which it is not the leader, it should sent back a response with an error code immediately. Right now, I see a produce request timing out in LogRecoveryTest since the producer never gets a response. It doesn't break the test since the producer retries, but it is adding unnecessary delay.
8.2. In quite a few places, we log an error and then retry, or log an error which is actually meant to be a warning. Due to this, it is hard to spot real errors during testing. We can probably fix this too as part of KAFKA-402.
8. Found a NPE while running ReplicaFetchTest. Would like to file a bug for it
9. Controller bugs
9.1. Unit tests fail intermittently due to NoSuchElementException thrown by testZKSendWithDeadBroker() in KafkaController. Due to this, the zookeeper server doesn't shut down and rest of the unit tests fail. The root cause is absence of a broker id as a key in the java map. I think Scala maps should be used as it forces the user to handle invalid values cleanly through Option variables.
9.2. ControllerBasicTest tests throw tons of zookeeper warnings that complain a client not cleanly closing a zookeeper session. The sessions are forcefully closed only when the zookeeper server is shutdown. These warnings should be paid attention to and fixed. LogTest is also throwing similar zk warnings.
9.3. Since controller bugs were not in the critical path to getting replication-with-failures to work, I'd like to file another JIRA to fix it.
If the above future work sounds good, I will go ahead and file the JIRAs
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420734#comment-13420734 ]
Jun Rao commented on KAFKA-350:
-------------------------------
Thanks for patch v2. Some comments:
20. PartitionMetadata: Could we make getLeaderRequest() private?
21. DefaultEventHandler.partitionAndCollate(): Is it necessary for this method to return an Option? It seems that if this method hits any exception, it's simpler to just pass on the exception and let the caller deal with it.
22. LogSegment: Ideally this class should only deal with offsets local to the segment. Global offsets are only used at the Log level. So it's probably better to use local offset in the input of truncateUpto(). Similarly, we probably don't need endOffset since it returns global offset. If we do want to keep it, it probably should be named globalEndOffset.
23. Log: Since we removed the HW check in FileMessageSet.read. We will need to add a guard in Log.read() so that we only expose messages up to HW to the consumer.
24. LogManager: remove unused import
25. ControllerChannelManager.removeBroker: the error message is not correct since this method is called in places other than shutdown too.
26. system_test/single_host_multi_brokers/bin/run-test.sh: We should remove comments saying "If KAFKA-350 is fixed".
27. Unit tests pass on my desktop but fail on my laptop at the following test consistently. Without the patch, unit tests pass on my laptop too. This seems to be due to the change in ZooKeeperTestHarness.
[info] Test Starting: testFetcher(kafka.consumer.FetcherTest)
[error] Test Failed: testFetcher(kafka.consumer.FetcherTest)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:31)
at kafka.consumer.FetcherTest.kafka$integration$KafkaServerTestHarness$$super$setUp(FetcherTest.scala:35)
at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:35)
at kafka.consumer.FetcherTest.setUp(FetcherTest.scala:57)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: (was: kafka-350-v1.patch)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Improve the high water mark
maintenance to store high watermarks for all partitions in a single file on
disk
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287007#comment-13287007 ]
Neha Narkhede commented on KAFKA-350:
-------------------------------------
Handle the logic of deleting segments that have start offset > highwatermark during log recovery.
> Improve the high water mark maintenance to store high watermarks for all partitions in a single file on disk
> ------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
>
> KAFKA-46 introduced per partition leader high watermarks. But it stores those in one file per partition. A more performant solution would be to store all high watermarks in a single file on disk
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421601#comment-13421601 ]
Neha Narkhede commented on KAFKA-350:
-------------------------------------
21. It is recoverable when there is auto create topic enabled on the server. Until we get rid of auto create, I think we can keep this.
30. Changed it to be non-negative and non-zero
Committing it with these changes.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch, kafka-350-v4.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-350) Improve the high water mark
maintenance to store high watermarks for all partitions in a single file on
disk
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede reassigned KAFKA-350:
-----------------------------------
Assignee: Neha Narkhede
> Improve the high water mark maintenance to store high watermarks for all partitions in a single file on disk
> ------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
>
> KAFKA-46 introduced per partition leader high watermarks. But it stores those in one file per partition. A more performant solution would be to store all high watermarks in a single file on disk
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420756#comment-13420756 ]
Jay Kreps commented on KAFKA-350:
---------------------------------
1.1, 1.2, 2.1, 2.2 Awesome!
2.3 Hmm, well so but that doesn't make sense because we would always pop the socket timeout and fully close out the connection, which is not good. Basically the socket timeout would ALWAYS go off before the server-side request processing timeout due to the network latency both ways. I think that timeout should just be for "emergencies". I agree it is a bit of a muddle now that we have two.
7.2 Hmm, but that changed ALL IllegalStateExceptions to KafkaException. IllegalStateException is used to say "this should not be possible". For example in SocketServer those checks are in the form of assertions. I like the idea of KafkaException, and I think we should make all the other exceptions extend it to ease error handling on the client, but I don't think we should get rid of IllegalStateException or IllegalArgumentException, they are informative.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418065#comment-13418065 ]
Jay Kreps commented on KAFKA-350:
---------------------------------
This is a pretty hard to review due to the large number of changes and also because I don't know some of this code well.
A lot of things like bad logging/naming that I think you could probably catch just perusing the diff.
Log:
- Log should not know about hw right? We seem to be adding more hw stuff there?
- This adds a getHW() that just returns a private val, why not make the val public? Please fix these. Regardless of cleanup being done get/set methods have been against the style guide for a long time, lets not add more. Ditto getEndOffset() which in addition to being a getter is inconsistent with Log.logEndOffset
- There is debug statement in a for loop in Log.scala that needs to be removed
- I don't understand the difference between nextAppendOffset and logEndOffset. Can you make it clear in the javadoc and explain on why we need both of these. Our public interface to Log is getting incredibly complex, which is really sad so I think we should really think through deeply what is added here and why.
- The javadoc on line 138 of Log.scala doesn't match the style of javadoc for the preceeding 5 variables.
- Does making isr.keep.in.sync.time.ms more stringent actually make sense? 10 seconds is pretty tight. I think what you are saying is that every server bounce will introduce 30 seconds of latency. But I think that is kind of a weakness of our design. If we lower that timeout we may just get spurious dropped replicas, no?
- Can we change the name of isr.keep.in.sync.time.ms to replica.max.lag.time.ms?
- Good point about the socket timeouts. We can't set socket timeout equal to request timeout, though, as there may be a large network latency. I recommend we just default the socket timeout to something large (like 10x the request timeout), and throw an exception if it is less than the request timeout (since that is certainly wrong). I don't think we should be using the socket timeout except as an emergency escape for a totally hung broker now that we have the nice per-request timeout.
- Can we change producer.request.ack.timeout.ms to producer.request.timeout.ms so it is more intuitive? I don't think the word "ack" is going to be self-explanatory to users.
SocketServer.scala
- Please remove: info("Shut down acceptor completed")
- Is there a reason to add the port into the thread name? That seems extremely odd...is it to simplify testing where there are multiple servers on a machine?
- Why is it a bug for processNewResponses() to happen while a shutdown is occuring. I don't think that is a bug. That is called in the event loop. It is the loop that should stop, no? Is there any ill effect of this?
- Good catch on the infinite loop
System testing
- I think we should fix the performance test hacks. The performance tool is critical. I have watched this play out before. No one ever budgets time for making the performance test stuff usable and then it just gets re-written umpteen times and never does what is needed. Most of these are just a matter of some basic cleanup and adding options. Let's work clean.
AdminUtils
- I don't understand the change in getTopicMetaDataFromZK
Replica.scala
- Can you remove trace("Returning leader replica for leader " + leaderReplicaId) unless you think it is of value going forward
ErrorMapping.scala
- getMaxErrorCodeValue - seems to be recomputed for each TopicMetadata. Let's get rid of this, I don't think we need it. We already have an unknown entry in the mapping, we should use that and get rid of the Utils.getShortInRange
- If we do want to keep it, fix the name
- We should really give a base class KafkaException to all exceptions so the client can handle them more easily
- Instead of having the client get an IllegalArgumentException we should just throw new KafkaException("Unknown server error (error code " + code + ")")
- The file NoLeaderForPartitionException seems to be empty now, I think you meant to delete it
ConsoleConsumer
- What does moving those things into a finally block do? We just caught all exceptions...
FileMessageSet
- You added more log spam. Please format it so it is intelligible to someone not working on the code or remove: debug("flush size:" + sizeInBytes())
- Ditto info("recover upto size:" + sizeInBytes())
BrokerPartitionInfos is a really weird class
DefaultEventHandler
- This seems to have grown into a big glump of proceedural logic.
- Inconsistent spacing of parens should be cleaned up
- partitionAndCollate is extemely complex
- Option[Map[Int, Map[(String, Int), Seq[ProducerData[K,Message]]]]]
ZkUtils
- LeaderExists class needs a name that is a noun
- Also we have a class with the same name in TopicMetadata
I really like TestUtils.waitUntilLeaderIsElected. God bless you for not adding sleeps. We should consider repeating this pattern for other cases like this.
ZooKeeperTestHarness.scala
- Can we replace the thread.sleep with a waitUntilZkIsUp call?
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: (was: kafka-350-v2.patch)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v3.patch
Thanks for the review !
Jun's comments -
20. Made getLeaderRequest and getLogSegmentMetadataRequest private
21. partitionAndCollate() is used by dispatchSerializedData() API. The contract of dispatchSerializedData() is to return the list of messages that were not sent successfully so that they can be used on the next retry. Now, if partitionAndCollate() fails, we need dispatchSerializedData to return the entire input list of messages. If something else fails after partitionAndCollate, we need to return a filtered list of messages back. To handle this, partitionAndCollate returns an Option. I agree that this class has very complex code that needs a cleanup. Have filed KAFKA-409 for that.
22. We need to know the absolute end offset of the segment for truncation during make follower state change. Have changed the name of the API to absoluteEndOffset.
23. We have KAFKA-376 filed to ensure that the consumer sees data only after the HW. This patch exposes data upto log end offset to replicas as well as consumers. So FileMessageSet.read() exposes all data upto log end offset.
24. Removed unused import.
25. Hmm, fixed the error message
26. Removed the TODOs related to KAFKA-350
27. Introduced the connection timeout and session timeout variables in the test harness and set it to 6s.
Jay's comments -
2.3 So we talked about this offline, stating it here so that others can follow. There are 2 choices here -
2.3.1. One is to keep request timeout = socket timeout. This would make us fully dependent on the socket for the timeout. So, any requests that took longer than that on the server for some reason (GC, bugs etc), would throw SocketTimeoutException on the producer client. The only downside to this is that the producer client wouldn't know why it failed or whether it got written to 0 or 1 or n replicas. Al though, this should be very rare and in the normal case, the server process the request and be able to send the response within the timeout. This is assuming the timeout value was set counting the network delay as well. So if the request is travelling cross-colo, it is expected that you set the timeout keeping in mind the cross-colo network latency.
2.3.2. The second choice is to set the socket timeout >> request timeout. This would ensure that in some rare failure cases on the server, the producer client would still be able to get back a response (most of the times) with a descriptive error message explaining the cause of the failure. However, it would also mean that under some failures like (server side kernel bug crashing the server etc), the timeout would actually be the socket timeout, which is usually set to a much higher value. This can confuse users who might expect the timeout to be the request timeout. Also, having two different timeouts also seems to complicate the guarantees provided by Kafka
2.3.3. I personally think option 1 is simpler and provides no worse guarantees than option 2. This patch just sets the socket timeout to be the request timeout.
2.4 Yeah, I think differentiating between “this should not be possible” and “this should not be possible in Kafka” is a little tricky. On one hand, it seems nicer to know that any exception thrown by Kafka will either be KafkaException or some exception that extends KafkaException. On the other hand, some states are just impossible and must be treated like assertions, for example, a socketchannel key that is not readable or writable or even valid. And in such cases, it might be slightly more convenient to have IllegalStateException. And I'm not sure I know the right answer here. I took a pass over all the IllegalStateException usages and converted the ones I think should be KafkaException, but I might not have done it in the best way possible.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Description: KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures (was: KAFKA-46 introduced per partition leader high watermarks. But it stores those in one file per partition. A more performant solution would be to store all high watermarks in a single file on disk)
Summary: Enable message replication in the presence of controlled failures (was: Improve the high water mark maintenance to store high watermarks for all partitions in a single file on disk)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v2.patch
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421590#comment-13421590 ]
Jun Rao commented on KAFKA-350:
-------------------------------
Thanks for patch v4. We are almost there.
21. In partitionAndCollate(), UnknownTopicException doesn't seem to recoverable.
30. It doesn't look like the broker handles requestTimeoutMs of 0 through RequestPurgatory. It's probably simpler if we just require timeout to be positive. If someone wants infinite timeout, MAX_INT can be used instead.
If those issues are addressed, the patch can be committed without another round of review.
34. This seems to only happen on my laptop. Will file another jira.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch, kafka-350-v4.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419280#comment-13419280 ]
Guozhang Wang commented on KAFKA-350:
-------------------------------------
Would it be better to have a general read/write formatting for boolean, like writeBoolean/readBoolean just like writeShortString/readShortString? Then we do not need to specify, for example, in TopicMetadata:
sealed trait LeaderRequest { def requestId: Byte }
case object LeaderExists extends LeaderRequest { val requestId: Byte = 1 }
case object LeaderDoesNotExist extends LeaderRequest { val requestId: Byte = 0 }
which would need to be done for every request/response that has a boolean.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v1.patch
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Status: Patch Available (was: In Progress)
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-350:
--------------------------------
Attachment: kafka-350-v2.patch
Thanks for the review, Jay ! Here is another patch fixing almost all review comments -
1. Log
1. Refactoring of the hw logic is part of KAFKA-405. It is moved to a new file HighwaterMarkCheckpoint and is controlled by the ReplicaManager. The Log does not and should not know about high watermarks.
2. Getter/setter for hw is removed as part of KAFKA-405 anyways.
3. Agree with you on the weak public interface, especially Log needs a cleanup. I think you attempted that as part of KAFKA-371. I've cleaned up quite a few things as part of KAFKA-405 and this patch. Nevertheless, fixed the nextAppendOffset as part of this patch. It is not required when we have logEndOffset. Also, removed the getEndOffset from FileMessageSet. Added endOffset() API to the LogSegment in addition to a size() API. This is useful during truncation of the log based on high watermark.
4. Fixed the javadoc and removed the debug statement.
2. Config options
1. The isr.keep.in.sync.time.ms set to 10 seconds is also very lax. A healthy follower should be able to catch up in 100s of milliseconds even with a lot of topics, with a worst case of maybe 4-5 seconds. We will know the latency better when we run some large scale tests. But yeah, the issue with the system test is independent of what the right value should be. I was just explaining how I discovered this issue. :)
2. Good point about renaming it to replica.max.lag.time.ms. Also changed isr.keep.in.sync.bytes to replica.max.lag.bytes.
3. Since the producer does a blocking read on the socket, the socket timeout cannot be greater than the request timeout. If it is, then the request timeout guarantee would be violated, no ?
3. SocketServer.scala
1. Removed info log statement
2. Yes, wanted to simplify testing/debugging (thread dumps) when there are multiple servers on one machine. Not sure if this is the best way to do that.
3. processNewResponses() doesn't have to process outstanding requests during shutdown. It can shutdown by ignoring them and those requests will timeout anyways. But yes, good point about the event loop doing it instead. Fixed it.
4. System testing
1. Fixed the hacky change. I still need ProducerPerformance needs a complete redo. Filed bug KAFKA-408 to do that.
5. AdminUtils
1. Need this change to return appropriate error codes for topic metadata request. Without this, all produce requests are timing out while fetching metadata, since in the system test, at any point of time, one replica is always down.
6. Partition.scala
1. Removed the trace statement.
7. ErrorMapping
1. Removed getMaxErrorCode and its usage from ErrorMapping
2. Good point. Introduced a new class KafkaException and converted IllegalStateException and IllegalArgumentException to it.
3. NoLeaderForPartitionException is in fact marked as deleted in the patch
8. ConsoleConsumer
1. Good point. The finally block there didn't really make any sense.
9. FileMessageSet
1. I haven't added the log statements in this patch, we always had it.
2. The purpose of adding that was help with debugging producer side queue full exceptions. We turned on DEBUG to see what the server side flush sizes and latencies were. I think these statements were added when we didn't have an ability to monitor these. However, whoever added the log flush monitoring maybe forgot to remove these statements. I removed it in this patch.
3. Removed the “recover upto” log statement too
10. DefaultEventHandler
1. Agree with the unwieldly procedural code. Filed bug KAFKA-409 to clean this up.
11. ZkUtils
1. Renamed LeaderExists -> LeaderExistsListener
12. ZookeeperTestHarness
1. The sleep is actually not required. The EmbeddedZookeeper constructor returns only after the zk server is completely started.
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (KAFKA-350) Improve the high water mark
maintenance to store high watermarks for all partitions in a single file on
disk
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on KAFKA-350 started by Neha Narkhede.
> Improve the high water mark maintenance to store high watermarks for all partitions in a single file on disk
> ------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
>
> KAFKA-46 introduced per partition leader high watermarks. But it stores those in one file per partition. A more performant solution would be to store all high watermarks in a single file on disk
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-350) Enable message replication in the
presence of controlled failures
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421235#comment-13421235 ]
Jun Rao commented on KAFKA-350:
-------------------------------
Thanks for patch v3.
For 21, my point is that the exceptions that can be thrown in partitionAndCollate() are non-recoverable and therefore retries won't help. partitionAndCollate() won't throw NoBrokersForPartitionException since only BrokerPartitionInfo.updateInfo can throw such an exception and updateInfo is not called here. partitionAndCollate() throws InvalidPartitionException that indicates a wrong partition of a topic. If we just retry, we will hit the same exception and fail again. It's simpler to just throw an exception and treat it as a failed case. Ditto for other exceptions that partitionAndCollate() may throw.
A few new comments:
30. SyncProducerConfig.requestTimeoutMs: We should make sure that the value is a positive integer and change the comment accordingly.
31. IteraterTemplate: The two KafkaExceptions are better reverted to IllegalStateException.
32. ProducerPerformance: We should remove socketTimeoutMsOpt.
33. SocketServer: no need to import illegalStateException
34. I got the following exception when running system_test/single_host_multi_brokers/bin/run-test.sh
2012-07-24 00:22:51 cleaning up kafka server log/data dir
2012-07-24 00:22:53 starting zookeeper
2012-07-24 00:22:55 starting cluster
2012-07-24 00:22:55 starting kafka server
2012-07-24 00:22:55 -> kafka_pids[1]: 75282
2012-07-24 00:22:55 starting kafka server
2012-07-24 00:22:55 -> kafka_pids[2]: 75286
2012-07-24 00:22:55 starting kafka server
2012-07-24 00:22:55 -> kafka_pids[3]: 75291
2012-07-24 00:22:57 creating topic [mytest] on [localhost:2181]
creation failed because of org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409)
at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:363)
at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:80)
at kafka.admin.CreateTopicCommand$.createTopic(CreateTopicCommand.scala:86)
at kafka.admin.CreateTopicCommand$.main(CreateTopicCommand.scala:73)
at kafka.admin.CreateTopicCommand.main(CreateTopicCommand.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1277)
at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99)
at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416)
at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 7 more
> Enable message replication in the presence of controlled failures
> -----------------------------------------------------------------
>
> Key: KAFKA-350
> URL: https://issues.apache.org/jira/browse/KAFKA-350
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Attachments: kafka-350-v1.patch, kafka-350-v2.patch, kafka-350-v3.patch
>
>
> KAFKA-46 introduced message replication feature in the absence of server failures. This JIRA will improve the log recovery logic and fix other bugs to enable message replication to happen in the presence of controlled server failures
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira