[jira] [Comment Edited] (KAFKA-5621) The producer should retry expired batches when retries are enabled

["Sumant Tambe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100349#comment-16100349 ] 

Sumant Tambe edited comment on KAFKA-5621 at 7/25/17 4:55 PM:
--------------------------------------------------------------

[~ijuma], [~apurva], It looks like this proposal will change the notional accumulator timeout from request.timeout.ms to  (r.t.m * retries). The bound on the send path of a record from the point when send returns will be up to linger.ms + r.t.m * retries + retries * (r.t.m. + retry.backoff.ms). So time of the send failure notification could potentially double as two full retry cycles will be applied. 

I prefer an explicit name to the accumulator timeout (we proposed batch.expiry.ms) but I'm well aware that adding new configs makes things harder for the enduser. I prefer to avoid overloading configs to do multiple jobs. That's confusing too. Applying retries to accumulator is not intuitive to me. It seems to jack up accumulator time by a multiple. 

Having said that, if there is strong agreement over not adding a new config, I'm fine with the proposed trick here.


was (Author: sutambe):
[~ijuma], [~apurva], It looks like this proposal will change the notional accumulator timeout from request.timeout.ms (r.t.m) to  (r.t.m * retries). The bound on the send path of a record from the point when send returns is up to linger.ms + r.t.m * retries + retries * (r.t.m. + retry.backoff.ms). So time of the send failure notification could potentially double as two full retry cycles will be applied. 

I prefer an explicit name to the accumulator timeout (we proposed batch.expiry.ms) but I'm well aware that adding new configs makes things harder for the enduser. I prefer to avoid overloading configs to do multiple jobs. That's confusing too. Applying retries to accumulator is not intuitive to me. It seems to jack up accumulator time by a multiple. 

Having said that, if there is strong agreement over not adding a new config, I'm fine with the proposed trick here.

> The producer should retry expired batches when retries are enabled
> ------------------------------------------------------------------
>
>                 Key: KAFKA-5621
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5621
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Apurva Mehta
>             Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is raised to the user.
> It might be better the producer to retry the expired batch rather up to the configured number of retries. This is more intuitive from the user's point of view. 
> Further the proposed behavior makes it easier for applications like mirror maker to provide ordering guarantees even when batches expire. Today, they would resend the expired batch and it would get added to the back of the queue, causing the output ordering to be different from the input ordering.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)