You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/03/16 19:18:15 UTC
[GitHub] [kafka] yatsukav opened a new pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
yatsukav opened a new pull request #10333:
URL: https://github.com/apache/kafka/pull/10333
A large number of topic-partitions on one broker causes burst of host's packets/sec metric. This pull request allow disable TCP_NODELAY socket option via Kafka Config.
Big amount topic-partitions per broker raise enormous packets count. For example 30k topic-partitions under load per 4 broker spawn ~150k packets/sec. With disabled TCP_NODELAY this value reduced to ~3k packets/sec.
More about how to reproduce problem and result after solving in JIRA: https://issues.apache.org/jira/browse/KAFKA-12481
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r627914726
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -741,6 +743,7 @@ object KafkaConfig {
"then controller will use \"broker1.example.com:9094\" with security protocol \"SSL\" to connect to the broker.\n" +
"If not explicitly configured, the default value will be null and there will be no dedicated endpoints for controller connections."
+ val SocketTcpNoDelayDoc = "Enable the Nagle algorithm (TCP_NODELAY) on broker sockets."
Review comment:
TCP_NODELAY disables the nagle algorithm, doesn't it? I think true is a bit misleading then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r595513484
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -391,6 +392,7 @@ object KafkaConfig {
val AdvertisedListenersProp = "advertised.listeners"
val ListenerSecurityProtocolMapProp = "listener.security.protocol.map"
val ControlPlaneListenerNameProp = "control.plane.listener.name"
+ val SocketNagleDisableProp = "socket.nagle.disable"
Review comment:
I think I would call this `socket.tcp.no.delay`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#issuecomment-800568236
Nice find. Since this introduces a new config, it would require a KIP:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r627914726
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -741,6 +743,7 @@ object KafkaConfig {
"then controller will use \"broker1.example.com:9094\" with security protocol \"SSL\" to connect to the broker.\n" +
"If not explicitly configured, the default value will be null and there will be no dedicated endpoints for controller connections."
+ val SocketTcpNoDelayDoc = "Enable the Nagle algorithm (TCP_NODELAY) on broker sockets."
Review comment:
TCP_NODELAY disables the nagle algorithm, doesn't it? I think this is a bit misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r627914726
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -741,6 +743,7 @@ object KafkaConfig {
"then controller will use \"broker1.example.com:9094\" with security protocol \"SSL\" to connect to the broker.\n" +
"If not explicitly configured, the default value will be null and there will be no dedicated endpoints for controller connections."
+ val SocketTcpNoDelayDoc = "Enable the Nagle algorithm (TCP_NODELAY) on broker sockets."
Review comment:
I think TCP_NODELAY disables the nagle algorithm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] yatsukav commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
yatsukav commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r595557313
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -391,6 +392,7 @@ object KafkaConfig {
val AdvertisedListenersProp = "advertised.listeners"
val ListenerSecurityProtocolMapProp = "listener.security.protocol.map"
val ControlPlaneListenerNameProp = "control.plane.listener.name"
+ val SocketNagleDisableProp = "socket.nagle.disable"
Review comment:
I take `socket.nagle.disable` from librdkafka project. But ok, I rename it to `socket.tcp.no.delay`. Perhaps this is clearer. Thank you for review.
I request KIP creating permission on dev@kafka.apache.org.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] yatsukav commented on a change in pull request #10333: KAFKA-12481: Add socket.nagle.disable config property
Posted by GitBox <gi...@apache.org>.
yatsukav commented on a change in pull request #10333:
URL: https://github.com/apache/kafka/pull/10333#discussion_r595557313
##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -391,6 +392,7 @@ object KafkaConfig {
val AdvertisedListenersProp = "advertised.listeners"
val ListenerSecurityProtocolMapProp = "listener.security.protocol.map"
val ControlPlaneListenerNameProp = "control.plane.listener.name"
+ val SocketNagleDisableProp = "socket.nagle.disable"
Review comment:
I take `socket.nagle.disable` from librdkafka project. But ok, I rename it to `socket.tcp.no.delay`. Perhaps this is clearer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org