You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/20 03:20:59 UTC
[GitHub] [incubator-seatunnel] EricJoy2048 opened a new issue, #2801: [Feature][Connector-V2 E2E] Data consistency test process design
EricJoy2048 opened a new issue, #2801:
URL: https://github.com/apache/incubator-seatunnel/issues/2801
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
We need support data consistency test in connector v2 e2e. I have some idea about it and welcome everyone to discuss.
#### Fake Source
If we want to test the data consistency of a sink connector, We can use the `Fake Source` connector. The Fake Source Connector support define row numbers and Primary key fields in the feature. Defile Primary key fields is useful to test exactly-once sink which implement exactly-once by Idempotent write data. If we can simulate task failure and then restore task, We can complete the data consistency test.
#### How to simulate task failure and restore task.
I think we can use `Fake Source` connector to simulate task failure too. We can add some active triggering failure function in Fake Source. To ensure Fake Source can support read playback, the Fake Source need support snapshot too.
#### How to check data
We can check the rows that wrote in sink.
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2801: [Feature][Connector-V2 E2E] Data consistency test process design
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2801:
URL: https://github.com/apache/incubator-seatunnel/issues/2801#issuecomment-1251798915
@getChan @2013650523 @531651225 @laglangyue @legendtkl @leo65535 @lhyundeadsoul @hailin0 @Hisoka-X @ashulin @ic4y @TyrantLucifer @iture123 and all people who may be interested in this question, Do you have any suggestions？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2801: [Feature][Connector-V2 E2E] Data consistency test process design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2801:
URL: https://github.com/apache/incubator-seatunnel/issues/2801#issuecomment-1316078610
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #2801: [Feature][Connector-V2 E2E] Data consistency test process design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #2801: [Feature][Connector-V2 E2E] Data consistency test process design
URL: https://github.com/apache/incubator-seatunnel/issues/2801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2801: [Feature][Connector-V2 E2E] Data consistency test process design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2801:
URL: https://github.com/apache/incubator-seatunnel/issues/2801#issuecomment-1286309735
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org