You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mas-chen (via GitHub)" <gi...@apache.org> on 2024/01/17 06:16:58 UTC
[PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
mas-chen opened a new pull request, #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79
…ith consistency and cleanup notify no more splits to ensure it is sent
I have spent the last few days troubleshooting this and found that most issues are related to the bounded mode tests. Issues:
1. Test records were not being produced to Kafka. I added the Kafka settings and refactored to the API to ensure all records are sent to all brokers.
2. When I fixed this, I ran into some tests that would exceed the 50 minute CI. This is because the the no more splits signal was not sent properly. I refactored the code a bit to make it clearer.
At this commit, the tests are quite stable. 6/7 CI runs pass.
<img width="329" alt="Screenshot 2024-01-16 at 10 13 43 PM" src="https://github.com/apache/flink-connector-kafka/assets/6834335/c0fae8e0-f8bc-4ad3-8075-ede06a27ef08">
In one CI run that failed, I did see that there was a Flink error, indicating a lost source event. In separate commits, there were other instances that exceeded 50 minutes but this can be solved separately, see https://issues.apache.org/jira/browse/FLINK-34127.
cc: @mxm @tzulitai @MartijnVisser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "mas-chen (via GitHub)" <gi...@apache.org>.
mas-chen commented on PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#issuecomment-1896211992
@MartijnVisser which thing to fix are you referring to? I think this PR is ready as-is if it blocking your other work. The ticket I'll need a few days to look into, probably get to it at the end of the week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#issuecomment-1895549968
> Can we put this under the original JIRA issue? I think that should be ok since we haven't release in the meantime.
Yeah that makes sense for me too, as a follow-up to actually complete the Jira
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "mas-chen (via GitHub)" <gi...@apache.org>.
mas-chen commented on code in PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#discussion_r1454686838
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumerator.java:
##########
@@ -212,32 +214,27 @@ public void start() {
private void handleNoMoreSplits() {
if (Boundedness.BOUNDED.equals(boundedness)) {
- enumContext.runInCoordinatorThread(
Review Comment:
No need to use this API. The code already runs in coordinator thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#issuecomment-1895470098
The only thing I'm wondering about is that we actually should file a Jira for this; otherwise LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#issuecomment-1895529046
Can we put this under the original JIRA issue? I think that should be ok since we haven't release in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-32416] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-32416] Fix flaky tests by ensuring test utilities produce records w… [flink-connector-kafka]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #79:
URL: https://github.com/apache/flink-connector-kafka/pull/79#issuecomment-1896516685
> which thing to fix are you referring to?
I misread https://github.com/apache/flink-connector-kafka/pull/79#discussion_r1454686838 as that you still wanted to fix something 🙈
I'll merge it after this CI run, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org