Re: "client" and requests in request.timeout.ms

[Guozhang Wang <wa...@gmail.com>]

Hello Stevo,

Thanks for your inputs. I think your comments can be further categorized
into the following points:

1. Is config request.timeout.ms leaky of internal implementations?

Your observation that "request.timeout.ms" is related only to NetworkClient
is right, but I think that does not mean we should not expose this config
to users, just like "send.buffer.bytes" and "receive.buffer.bytes" are also
for the internal NetworkClient.

For client configs we have IMPORTANCE levels for such configs, and internal
configs like these are usually marked LOW meaning that most of time users
would not need to modify its default values.


2. NetworkClient is actually used both in broker and clients. Is the
javadoc out dated?

Yes I agree. Current NetworkClient is used for controller-to-broker
communication including shutdown request sending, and replica fetcher
consumer request sending. When request timed out the corresponding
destination nodes are marked as disconnected, and it's the caller's
responsibility to re-send the request and try to reconnect to the node.

Could you file a JIRA to update the javadoc of NetworkClient?


3. Could we consider having a different request timeout for different types
of requests?

The community have been discussing about this (e.g.
https://github.com/apache/kafka/pull/3503). Personally I think it makes
sense to have a longer request timeout for some request types which may be
parked by the broker at the purgatory for some long time before responding,
like Produce / Consume / JoinGroup requests, etc.

Guozhang




On Fri, Jul 28, 2017 at 1:12 AM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Apache Kafka community,
>
> In Consumer, Producer, AdminClient and Broker configuration documentation
> there's a common config property, request.timeout.ms, with common
> description part being:
> "The configuration controls the maximum amount of time the client will wait
> for the response of a request. If the response is not received before the
> timeout elapses the client will resend the request if necessary or fail the
> request if retries are exhausted."
>
> If I'm not mistaken "client" term in all the different request.timeout.ms
> config property descriptions actually refers to NetworkClient, which is
> kind of leaky internal abstraction. It seems there's no mentioning of
> NetworkClient on the Kafka documentation page. By it's javadoc
> NetworkClient is:
> "A network client for asynchronous request/response network i/o. This is an
> internal class used to implement the user-facing producer and consumer
> clients."
> Since it's considered to be internal class maybe it could be moved in
> "internal" package as other internal classes.
> More importantly NetworkClient javadoc (second sentence) is not entirely
> true, since NetworkClient is used on the broker side too, e.g. to exchange
> controlled shutdown request/response, which IMO has nothing to do with
> "user-facing producer and consumer clients". Because NetworkClient
> abstraction is used on the broker side, there's request.timeout.ms config
> property not only for producer/consumer but also in broker configuration.
>
> Can somebody please verify if my understanding of the current situation is
> correct?
>
> There's no mentioning in the Kafka documentation about which requests will
> be affected by tuning each of the request.timeout.ms config properties, or
> how if at all are different request timeouts related.
>
> Specifically I'd like to lower producer/consumer request timeout, so
> user-facing client requests like Produce/Fetch/Metadata should be affected,
> but e.g. controlled shutdown requests on the broker side should not. I'm
> not sure whether broker side request timeout can be left unchanged or if
> there's combination/chain of client and broker side request/responses that
> are related so that the request timeout settings have to be kept in sync. I
> guess maybe client side Produce request and broker side replica Fetch form
> kind of a chain/dependency - depending on acks Produce cannot finish
> successfully until enough of replicas got the message. Producer's "
> request.timeout.ms" description explains relationship with Broker's "
> replica.lag.time.max.ms" (potential produce failure or duplication of
> messages due to retries being negative side-effects) but relationship with
> Broker's "request.timeout.ms" is not covered. Similarly Consumer's Fetch
> request to lead broker seems can only retrieve messages replicated to rest
> of ISR set, so there's again kind of dependency on replica Fetch, this time
> dependency has not so negative side-effect, it seems there could be more
> empty reads if Consumer request timeout is lower than Broker's which is a
> tradeoff, lower latency of individual requests vs lower load / number of
> requests.
>
> Is there a reason that the producer/consumer request timeout are set to
> same default value as request timeout default on the broker side?
>
> Additionally, is there are reason why there are no per-type-of-request
> timeout config properties? Does 30sec timeout really make sense for all
> kinds of requests, like user facing and internal coordination requests? New
> AdminClient request timeout is the only one it seems to have different,
> 120sec as default request timeout (old Scala AdminClient it seems has 5sec
> default request timeout).
>
> On a related note, even after looking at ServerSocketTest I'm not sure what
> happens with request handling on broker side when producer/consumer side
> timeout occurs - how will broker treat this, will it stop request
> processing and release any resources used, or will it continue to process
> the request and consume resources until request is full processed so when
> producer/consumer retries its timedout request, the broker side load will
> just multiply with number of retries?
>
> Kind regards,
> Stevo Slavic.
>



-- 
-- Guozhang