You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2023/01/10 07:09:04 UTC
[GitHub] [incubator-seatunnel] lightzhao opened a new pull request, #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
lightzhao opened a new pull request, #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906
When the checkpoint is not completed, if the flink job is killed or canceled, dirty files will be left under tmp and should be deleted.
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2)
2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [ ] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1378314551
Flink's SinkWriter does not declare the close method, which cannot be called when yarn kill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1377073648
> > Can you provide test cases?
>
> I have completed the test. When the checkpoint is not completed, I kill the flink job. The files in the temporary directory are deleted successfully.
Offer test case screenshot and write test cases, thanks.
BTW, if you choose offer test case screenshot, you need to provide all the steps to prove that this PR is valid and meaningful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on a diff in pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on code in PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#discussion_r1065463350
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/ParquetWriteStrategy.java:
##########
@@ -299,4 +299,8 @@ private Schema buildAvroSchemaWithRowType(SeaTunnelRowType seaTunnelRowType, Lis
MessageType seaTunnelRow = Types.buildMessage().addFields(types.toArray(new Type[0])).named("SeaTunnelRecord");
return schemaConverter.convert(seaTunnelRow);
}
+
+ @Override
+ public void close() {
+ }
Review Comment:
update complete.
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java:
##########
@@ -359,4 +359,8 @@ private void setDoubleVector(Object value, DoubleColumnVector doubleVector, int
throw new FileConnectorException(CommonErrorCode.ILLEGAL_ARGUMENT, errorMsg);
}
}
+
+ @Override
+ public void close() {
+ }
Review Comment:
update complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1376864666
@EricJoy2048 @Hisoka-X PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on a diff in pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on code in PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#discussion_r1065588415
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:
##########
@@ -328,6 +329,24 @@ public String getTargetLocation(@NonNull String seaTunnelFilePath) {
return tmpPath.replaceAll(BaseSinkConfig.NON_PARTITION + Matcher.quoteReplacement(File.separator), "");
}
+ /**
+ * Delete temporary files not caused by checkpoint.
+ * @param tmpFilePaths temporary files.
+ * @return void
+ */
+ public void deleteTmpFileWithNoCheckPoint(Set<String> tmpFilePaths){
+ try {
+ if (tmpFilePaths != null && tmpFilePaths.size() > 0) {
+ for (String filePath : tmpFilePaths){
+ fileSystemUtils.deleteFile(filePath);
+ log.info("delete no commit file:{}", filePath);
Review Comment:
complete modify.
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:
##########
@@ -328,6 +329,24 @@ public String getTargetLocation(@NonNull String seaTunnelFilePath) {
return tmpPath.replaceAll(BaseSinkConfig.NON_PARTITION + Matcher.quoteReplacement(File.separator), "");
}
+ /**
+ * Delete temporary files not caused by checkpoint.
+ * @param tmpFilePaths temporary files.
+ * @return void
+ */
+ public void deleteTmpFileWithNoCheckPoint(Set<String> tmpFilePaths){
+ try {
+ if (tmpFilePaths != null && tmpFilePaths.size() > 0) {
+ for (String filePath : tmpFilePaths){
+ fileSystemUtils.deleteFile(filePath);
+ log.info("delete no commit file:{}", filePath);
+ }
+ }
+ } catch (Exception e){
+ log.error("delete no commit file failed", e);
Review Comment:
complete modify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1378146425
> > > Can you provide test cases?
> >
> >
> > I have completed the test. When the checkpoint is not completed, I kill the flink job. The files in the temporary directory are deleted successfully.
>
> Offer test case screenshot and write test cases, thanks.
>
> BTW, if you choose offer test case screenshot, you need to provide all the steps to prove that this PR is valid and meaningful.
I still need to continue the test. I found that the close method cannot be called after yarn kill. This may have something to do with the flink mechanism. So far, there is still a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on a diff in pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
liugddx commented on code in PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#discussion_r1065440501
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/ParquetWriteStrategy.java:
##########
@@ -299,4 +299,8 @@ private Schema buildAvroSchemaWithRowType(SeaTunnelRowType seaTunnelRowType, Lis
MessageType seaTunnelRow = Types.buildMessage().addFields(types.toArray(new Type[0])).named("SeaTunnelRecord");
return schemaConverter.convert(seaTunnelRow);
}
+
+ @Override
+ public void close() {
+ }
Review Comment:
There's no operation here?
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java:
##########
@@ -359,4 +359,8 @@ private void setDoubleVector(Object value, DoubleColumnVector doubleVector, int
throw new FileConnectorException(CommonErrorCode.ILLEGAL_ARGUMENT, errorMsg);
}
}
+
+ @Override
+ public void close() {
+ }
Review Comment:
There's no operation here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FlechazoW commented on a diff in pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
FlechazoW commented on code in PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#discussion_r1065564635
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:
##########
@@ -328,6 +329,24 @@ public String getTargetLocation(@NonNull String seaTunnelFilePath) {
return tmpPath.replaceAll(BaseSinkConfig.NON_PARTITION + Matcher.quoteReplacement(File.separator), "");
}
+ /**
+ * Delete temporary files not caused by checkpoint.
+ * @param tmpFilePaths temporary files.
+ * @return void
+ */
+ public void deleteTmpFileWithNoCheckPoint(Set<String> tmpFilePaths){
+ try {
+ if (tmpFilePaths != null && tmpFilePaths.size() > 0) {
+ for (String filePath : tmpFilePaths){
+ fileSystemUtils.deleteFile(filePath);
+ log.info("delete no commit file:{}", filePath);
+ }
+ }
+ } catch (Exception e){
+ log.error("delete no commit file failed", e);
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1376867017
Can you provide test cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao commented on PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#issuecomment-1376897044
> Can you provide test cases?
I have completed the test. When the checkpoint is not completed, I kill the flink job. The files in the temporary directory are deleted successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FlechazoW commented on a diff in pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
FlechazoW commented on code in PR #3906:
URL: https://github.com/apache/incubator-seatunnel/pull/3906#discussion_r1065564394
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:
##########
@@ -328,6 +329,24 @@ public String getTargetLocation(@NonNull String seaTunnelFilePath) {
return tmpPath.replaceAll(BaseSinkConfig.NON_PARTITION + Matcher.quoteReplacement(File.separator), "");
}
+ /**
+ * Delete temporary files not caused by checkpoint.
+ * @param tmpFilePaths temporary files.
+ * @return void
+ */
+ public void deleteTmpFileWithNoCheckPoint(Set<String> tmpFilePaths){
+ try {
+ if (tmpFilePaths != null && tmpFilePaths.size() > 0) {
+ for (String filePath : tmpFilePaths){
+ fileSystemUtils.deleteFile(filePath);
+ log.info("delete no commit file:{}", filePath);
Review Comment:
What's the meaning of 'no commit file'? Why not use the 'temporary file ' or 'uncommitted file'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao closed pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
Posted by GitBox <gi...@apache.org>.
lightzhao closed pull request #3906: [Improve][connector-v2][HdfsFileSink]Delete temporary files not caused by checkpoint.
URL: https://github.com/apache/incubator-seatunnel/pull/3906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org