You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/05/20 07:48:02 UTC
[GitHub] [kafka] showuon opened a new pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
showuon opened a new pull request #10736:
URL: https://github.com/apache/kafka/pull/10736
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman merged pull request #10736: KAFKA-9295: revert session timeout to default value
Posted by GitBox <gi...@apache.org>.
ableegoldman merged pull request #10736:
URL: https://github.com/apache/kafka/pull/10736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #10736: KAFKA-9295: revert session timeout to default value
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-856345646
Hard to say. Also the logs are truncated. However, it seems there is some issue in the beginning that maybe some topic where not created?
```
[2021-05-27 08:36:11,424] WARN [Consumer clientId=KTable-FKJ-Multi-e7433918-5346-4bc4-b9b8-41bc75e16fa9-StreamThread-1-consumer, groupId=KTable-FKJ-Multi] Error while fetching metadata with correlation id 2 : {KTable-FKJ-Multi-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000017-topic=UNKNOWN_TOPIC_OR_PARTITION, KTable-FKJ-Multi-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000030-topic=UNKNOWN_TOPIC_OR_PARTITION, KTable-FKJ-Multi-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000009-topic=UNKNOWN_TOPIC_OR_PARTITION, KTable-FKJ-Multi-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000022-topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1092)
```
Note the timestamp of `08:36:11`.
After the truncated logs continue, the timestamp is `10:20:17` ... -- it's an almost 2h gap ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-850396020
OK, on second thought, I start to think the root cause of this flaky test is because the bug mentioned above. Increasing session timeout is just a workaround to avoid rebalancing caused data lost. I'll look into it further, and file a bug then. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-850092117
@ableegoldman , do you have any thought? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon edited a comment on pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
Posted by GitBox <gi...@apache.org>.
showuon edited a comment on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-850119016
1 more thing to add, because I can still see heartbeat expiration after increasing to 20 seconds:
```
reason: removing member KTable-FKJ-Multi-98311d03-31f2-4b20-bfcf-0cfc516741f5-StreamThread-1-consumer-06247087-0bc9-4f74-b0a4-a617c638e427 on heartbeat expiration
```
So, the session timeout increasing to 30 sec is to avoid this unexpected rebalancing happen, to make the test reliable. I think it makes sense. Just concerning the data lost issue mentioned above. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon edited a comment on pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
Posted by GitBox <gi...@apache.org>.
showuon edited a comment on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-850396020
OK, on second thought, I start to think the root cause of this flaky test is because the bug mentioned above. Increasing session timeout is just a workaround to avoid rebalancing caused data lost. I'll look into it further, and file a bug then. Any suggestions are welcomed.
Before the root cause got fixed, I think we can workaround it by increasing more session timeout. What do you think?
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #10736: KAFKA-9295: revert session timeout to default value
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-858384704
Thanks @mjsax ! I'll further investigate it and let you know~
@ableegoldman , since #10803 is already merged into trunk, now the default session timeout is 45 seconds. I think we can merge this PR to revert previous session timeout increasing workaround. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #10736: KAFKA-9295: revert session timeout to default value
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-856331849
Hey @vvcephei @bellemare , any thoughts on Luke's analysis above? I'm not super familiar with the FKJ so it's difficult for me to say whether this sounds like a possible data loss bug or not. That said, the symptoms do seem quite suspicious -- no other test is suffering from this degree of flakiness due to rebalancing, and even if the test does incur some rebalances here and there, it should still be able to process at least one record I would think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #10736: KAFKA-9295: increase session timeout to 30 seconds
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10736:
URL: https://github.com/apache/kafka/pull/10736#issuecomment-850119016
1 more thing to add, because I can still see
```
reason: removing member KTable-FKJ-Multi-98311d03-31f2-4b20-bfcf-0cfc516741f5-StreamThread-1-consumer-06247087-0bc9-4f74-b0a4-a617c638e427 on heartbeat expiration
```
So, the session timeout increasing is to avoid this unexpected rebalancing happen, to make the test reliable. I think it makes sense. Just concerning the data lost issue mentioned above. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org