You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "linyu003 (via GitHub)" <gi...@apache.org> on 2023/03/31 15:26:49 UTC
[GitHub] [incubator-seatunnel] linyu003 opened a new pull request, #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
linyu003 opened a new pull request, #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
[issues/4247](https://github.com/apache/incubator-seatunnel/issues/4247)
problems illustration: [issues/4247#issuecomment-1492074387](https://github.com/apache/incubator-seatunnel/issues/4247#issuecomment-1492074387)
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
* [x] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2)
2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [x] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] ic4y commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y commented on PR #4469:
URL: https://github.com/apache/seatunnel/pull/4469#issuecomment-1666808026
@linyu003 Please merge the latest dev branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] linyu003 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "linyu003 (via GitHub)" <gi...@apache.org>.
linyu003 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1505102082
The IT added by me failed in https://github.com/apache/incubator-seatunnel/actions/runs/4645826394/jobs/8246477241.
That's so strange since it run well on my local machine with ubuntu 22.04.
I am overwhelmed about this problem, help wanted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1502088089
> Great, I saw that you detailed the steps and solutions for the reproduce, It's great. Thanks for your contribution.
>
> @TyrantLucifer @hailin0 PTAL
waitting for ci check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1493310742
> > `ReflectionUtils.setField(transactionManager, "transactionStarted", true);` we need to additionally persist this transactionStarted flag in checkpoints as part of the KafkaCommitInfo metadata. And then set this flag appropriately to what is stored in the checkpoint. If it is an empty transaction, the transactionStarted flag should be set to false.
>
> I think that's a better way to fix the Problem 3, and tested well in my machine.
>
> When transactionStarted is set to false, KafkaProducer will close the transaction locally without sending the EndTxn request to kafka servers, so the problem will not appear.
>
> I have changed my pr. @lightzhao
great, please add e2e test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] linyu003 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "linyu003 (via GitHub)" <gi...@apache.org>.
linyu003 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1507737236
> > The IT added by me failed in https://github.com/apache/incubator-seatunnel/actions/runs/4645826394/jobs/8246477241. That's so strange since it run well on my local machine with ubuntu 22.04.
> >
> > I am overwhelmed about this problem, help wanted.
>
> Don't worry, I will help you restart it.
thank you so much.
The IT passed this time. stranged and surprising.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics] [seatunnel]
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #4469:
URL: https://github.com/apache/seatunnel/pull/4469#issuecomment-1833964915
waiting for ci passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics] [seatunnel]
Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on PR #4469:
URL: https://github.com/apache/seatunnel/pull/4469#issuecomment-1980280776
Waiting for this pr merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] linyu003 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "linyu003 (via GitHub)" <gi...@apache.org>.
linyu003 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1492826884
> `ReflectionUtils.setField(transactionManager, "transactionStarted", true);` we need to additionally persist this transactionStarted flag in checkpoints as part of the KafkaCommitInfo metadata. And then set this flag appropriately to what is stored in the checkpoint. If it is an empty transaction, the transactionStarted flag should be set to false.
I think that's a better way to fix the Problem 3, and tested well in my machine.
When transactionStarted is set to false, KafkaProducer will close the transaction locally without sending the EndTxn request to kafka servers, so the problem will not appear.
I will change my pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] linyu003 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "linyu003 (via GitHub)" <gi...@apache.org>.
linyu003 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1500909647
Sorry to be late, it took me some time to be familiar with the IT framework.
I have reverted unexpected change. @EricJoy2048
mvn spotless:apply is done @TyrantLucifer
E2E IT was added @lightzhao @EricJoy2048
IT failed before code change:
IT passed after code change:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1492892980
Use `mvn spotless:apply` to format code style before you submit pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1541476373
Please rebase from dev and push again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1505472986
> The IT added by me failed in https://github.com/apache/incubator-seatunnel/actions/runs/4645826394/jobs/8246477241. That's so strange since it run well on my local machine with ubuntu 22.04.
>
> I am overwhelmed about this problem, help wanted.
Don't worry, I will help you restart it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] ic4y closed pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y closed pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
URL: https://github.com/apache/seatunnel/pull/4469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1492850347
Please add e2e test case for this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on code in PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#discussion_r1155065452
##########
seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml:
##########
@@ -67,6 +67,11 @@
<artifactId>connector-console</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review Comment:
Please revert this. Otherwise, there will be a license issue.
##########
seatunnel-examples/seatunnel-flink-connector-v2-example/src/main/resources/examples/fake_to_kafka.conf:
##########
@@ -0,0 +1,34 @@
+env {
Review Comment:
Please revert this file. In fact we already ignore the file add in `seatunnel-examples` module from `.gitignore` file. I don't know why it seems not work in your pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #4469: [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4469:
URL: https://github.com/apache/incubator-seatunnel/pull/4469#issuecomment-1492848596
Great, I saw that you detailed the steps and solutions for the reproduce, It's great.
Thanks for your contribution.
@TyrantLucifer @hailin0 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug][Connector-v2][KafkaSink][Problems relating to commiting transactions under EXACTLY_ONCE semantics] [seatunnel]
Posted by "wu-a-ge (via GitHub)" <gi...@apache.org>.
wu-a-ge commented on PR #4469:
URL: https://github.com/apache/seatunnel/pull/4469#issuecomment-1880089628
When was the bug merged into dev?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org