You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/04/27 23:28:54 UTC
[GitHub] [kafka] ijuma opened a new pull request #8567: KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
ijuma opened a new pull request #8567:
URL: https://github.com/apache/kafka/pull/8567
After KIP-219, responses are sent immediately and we rely on a combination
of clients and muting of the channel to throttle. The result of this is that
we need to track `apiThrottleTimeMs` as an explicit value instead of
inferring it. On the other hand, we no longer need
`apiRemoteCompleteTimeNanos`.
Extend `BaseQuotaTest` to verify that throttle time in the request channel
metrics are being set. Given the nature of the throttling numbers, the test
is not particularly precise.
I included a few clean-ups:
* Pass KafkaMetric to QuotaViolationException so that the caller doesn't
have to retrieve it from the metrics registry.
* Inline Supplier in SocketServer (use SAM).
* Reduce redundant `time.milliseconds` and `time.nanoseconds`calls.
* Use monotonic clock in ThrottledChannel and simplify `compareTo` method.
* Simplify `TimerTaskList.compareTo`.
* Consolidate the number of places where we update `apiLocalCompleteTimeNanos`
and `responseCompleteTimeNanos`.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #8567: KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #8567:
URL: https://github.com/apache/kafka/pull/8567#discussion_r417665485
##########
File path: core/src/test/scala/integration/kafka/api/BaseQuotaTest.scala
##########
@@ -235,14 +240,29 @@ abstract class QuotaTestClients(topic: String,
numConsumed
}
- def verifyProduceThrottle(expectThrottle: Boolean, verifyClientMetric: Boolean = true): Unit = {
+ def verifyThrottleTimeRequestChannelMetric(apiKey: ApiKeys, metricNameSuffix: String,
Review comment:
Could this be private?
##########
File path: core/src/main/scala/kafka/server/KafkaApis.scala
##########
@@ -3012,17 +3010,24 @@ class KafkaApis(val requestChannel: RequestChannel,
private def sendResponseMaybeThrottle(request: RequestChannel.Request,
createResponse: Int => AbstractResponse,
onComplete: Option[Send => Unit] = None): Unit = {
- val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request)
- quotas.request.throttle(request, throttleTimeMs, sendResponse)
+ val throttleTimeMs = maybeRecordAndGetThrottleTimeMs(request)
+ quotas.request.throttle(request, throttleTimeMs, requestChannel.sendResponse)
sendResponse(request, Some(createResponse(throttleTimeMs)), onComplete)
}
private def sendErrorResponseMaybeThrottle(request: RequestChannel.Request, error: Throwable): Unit = {
- val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request)
- quotas.request.throttle(request, throttleTimeMs, sendResponse)
+ val throttleTimeMs = maybeRecordAndGetThrottleTimeMs(request)
+ quotas.request.throttle(request, throttleTimeMs, requestChannel.sendResponse)
sendErrorOrCloseConnection(request, error, throttleTimeMs)
}
+ private def maybeRecordAndGetThrottleTimeMs(request: RequestChannel.Request): Int = {
+ val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request, time.milliseconds())
+ println(s"api throttle ms $throttleTimeMs ${request.header} ${request.header.clientId}")
Review comment:
I guess this is not intended?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #8567: KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #8567:
URL: https://github.com/apache/kafka/pull/8567#discussion_r417670946
##########
File path: core/src/main/scala/kafka/server/KafkaApis.scala
##########
@@ -3012,17 +3010,24 @@ class KafkaApis(val requestChannel: RequestChannel,
private def sendResponseMaybeThrottle(request: RequestChannel.Request,
createResponse: Int => AbstractResponse,
onComplete: Option[Send => Unit] = None): Unit = {
- val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request)
- quotas.request.throttle(request, throttleTimeMs, sendResponse)
+ val throttleTimeMs = maybeRecordAndGetThrottleTimeMs(request)
+ quotas.request.throttle(request, throttleTimeMs, requestChannel.sendResponse)
sendResponse(request, Some(createResponse(throttleTimeMs)), onComplete)
}
private def sendErrorResponseMaybeThrottle(request: RequestChannel.Request, error: Throwable): Unit = {
- val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request)
- quotas.request.throttle(request, throttleTimeMs, sendResponse)
+ val throttleTimeMs = maybeRecordAndGetThrottleTimeMs(request)
+ quotas.request.throttle(request, throttleTimeMs, requestChannel.sendResponse)
sendErrorOrCloseConnection(request, error, throttleTimeMs)
}
+ private def maybeRecordAndGetThrottleTimeMs(request: RequestChannel.Request): Int = {
+ val throttleTimeMs = quotas.request.maybeRecordAndGetThrottleTimeMs(request, time.milliseconds())
+ println(s"api throttle ms $throttleTimeMs ${request.header} ${request.header.clientId}")
Review comment:
Yes, sorry, forgot to remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #8567: KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #8567:
URL: https://github.com/apache/kafka/pull/8567#discussion_r417674818
##########
File path: core/src/test/scala/integration/kafka/api/BaseQuotaTest.scala
##########
@@ -235,14 +240,29 @@ abstract class QuotaTestClients(topic: String,
numConsumed
}
- def verifyProduceThrottle(expectThrottle: Boolean, verifyClientMetric: Boolean = true): Unit = {
+ def verifyThrottleTimeRequestChannelMetric(apiKey: ApiKeys, metricNameSuffix: String,
Review comment:
Yes. I also restricted access to a few other methods in this class where possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8567: KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8567:
URL: https://github.com/apache/kafka/pull/8567#issuecomment-620887867
2 jobs passed, 1 unrelated flaky test failed:
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org