You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Joel Koshy (JIRA)" <ji...@apache.org> on 2011/08/04 00:17:27 UTC
[jira] [Updated] (KAFKA-75) Kafka mirror does not shutdown on IO
error
[ https://issues.apache.org/jira/browse/KAFKA-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-75:
----------------------------
Description:
The embedded consumer in the kafka mirror implementation uses the low-level Log api to create the replica. The append operation may fail and result in a corrupt log file, due to an IO error which is currently caught and ignored.
The proposed fix is to switch to using the high-level producer API to create the replica. Not only would this avoid the above issue, but it would also fit better with the current design of the replication enhancement for kafka (http://linkedin.jira.com/browse/KAFKA-23), since the low-level Log api is not replication-aware. Another advantage is that compression is exposed at the producer API-level. One caveat in this approach would be the following: the async producer drops events when its queue is full. This behavior is unsuitable for the embedded consumer, so we can expose a configuration option in the producer to allow for (queue-level) blocking semantics.
was:
The embedded consumer in the corp replica uses the low-level Log api to create the replica. The append operation may fail and result in a corrupt log file, due to an IO error which is currently caught and ignored.
The proposed fix is to switch to using the high-level producer API to create the replica. Not only would this avoid the above issue, but it would also fit better with the current design of the replication enhancement for kafka (http://linkedin.jira.com/browse/KAFKA-23), since the low-level Log api is not replication-aware. Another advantage is that compression is exposed at the producer API-level. One caveat in this approach would be the following: the async producer drops events when its queue is full. This behavior is unsuitable for the embedded consumer, so we can expose a configuration option in the producer to allow for (queue-level) blocking semantics.
Summary: Kafka mirror does not shutdown on IO error (was: Corp replica does not shutdown on IO error)
> Kafka mirror does not shutdown on IO error
> ------------------------------------------
>
> Key: KAFKA-75
> URL: https://issues.apache.org/jira/browse/KAFKA-75
> Project: Kafka
> Issue Type: Improvement
> Reporter: Joel Koshy
>
> The embedded consumer in the kafka mirror implementation uses the low-level Log api to create the replica. The append operation may fail and result in a corrupt log file, due to an IO error which is currently caught and ignored.
> The proposed fix is to switch to using the high-level producer API to create the replica. Not only would this avoid the above issue, but it would also fit better with the current design of the replication enhancement for kafka (http://linkedin.jira.com/browse/KAFKA-23), since the low-level Log api is not replication-aware. Another advantage is that compression is exposed at the producer API-level. One caveat in this approach would be the following: the async producer drops events when its queue is full. This behavior is unsuitable for the embedded consumer, so we can expose a configuration option in the producer to allow for (queue-level) blocking semantics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira