You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/29 13:31:10 UTC
[GitHub] [dolphinscheduler] sketchmind opened a new pull request, #10675: [Fix-10665] [S3] Fix s3 download method
sketchmind opened a new pull request, #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
fix s3 download error, which cause task failure
fix #10665
## Brief change log
* when overwrite param is true and dest file is exist, throw exception
* make dirs for ancestor directories of the source file path if it isn't exists, if create failed, throw exception
* To describe the error more clearly, change "the file isn`t exists" to specific error message
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1176959700
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [20 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.6% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915390535
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
> > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
>
> The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
You only need to `overwrite` this part to be consistent. The rest can remain the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> We should remove this return.
If so, It will result in false of overwrite not being processed essentially
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911569143
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
Review Comment:
Looks like you've solved it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911568551
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
Review Comment:
> Please format this line.
like this?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r910744602
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
+ }
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
byte[] readBuf = new byte[1024];
- int readLen = 0;
+ int readLen;
while ((readLen = s3is.read(readBuf)) > 0) {
fos.write(readBuf, 0, readLen);
}
} catch (AmazonServiceException e) {
- logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+ logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
throw new IOException(e.getMessage());
} catch (FileNotFoundException e) {
- logger.error("the file isn`t exists");
- throw new IOException("the file isn`t exists");
+ logger.error("the destination file {} isn`t exists", dstFilePath);
+ throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);
Review Comment:
Please log the stack of e in line 203 or directly throw it. And in fact, I think directly throw FileNotFoundException in 206 is better than IOException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911561303
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
Review Comment:
Please format this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1176960475
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [20 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.6% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915463868
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> It is useful when the user modifies the contents of the file and reruns the task.
Do you mean that the destination file should be overwritten no matter what the overwrite parameter is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> We should remove this return.
If so, It will result in false of overwrite not being processed on the market cap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179994987
> > > LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
> >
> >
> > Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code
>
> This should add a comment in the code to address the reason.
@SbloodyS Could you give me the code location of the rerun, I'll see if it has any effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1196783479
> Thanks for your contribution. This is a good beginning. Considering that it's your first contribution, I think we can get deep communication, you can contact me through mail or add WeChat(SbloodyS), when mail or added, please tell me who you are. I think I can help to familiar with the Dolphinscheduler if you meet with problems.
Thank you for your excellent suggestions and help during the submission process, hope the community is getting better and better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170089919
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179891348
> LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175686382
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [3 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [57.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [1.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915418399
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
> > > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
> > >
> > >
> > > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
> >
> >
> > You only need to `overwrite` this part to be consistent. The rest can remain the same.
>
> Wait a minute, let me have a look
> > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
> >
> >
> > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
>
> You only need to `overwrite` this part to be consistent. The rest can remain the same.
The code has been updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911563198
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
Review Comment:
> Please format this line.
The code is already the same as the latest one you gave
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177278414
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915487719
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using
`
if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) {
return false;
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914356488
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
Review Comment:
> I think we should delete it instead of throwing exception if `overwrite` is true.
Ok, by the way ,do I need to throw an error if the path is a directory and the overwrite if false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911560936
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
+ }
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
byte[] readBuf = new byte[1024];
- int readLen = 0;
+ int readLen;
while ((readLen = s3is.read(readBuf)) > 0) {
fos.write(readBuf, 0, readLen);
}
} catch (AmazonServiceException e) {
- logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+ logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
throw new IOException(e.getMessage());
} catch (FileNotFoundException e) {
- logger.error("the file isn`t exists");
- throw new IOException("the file isn`t exists");
+ logger.error("the destination file {} isn`t exists", dstFilePath);
+ throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);
Review Comment:
> Please log the stack of e in line 203 or directly throw it. And in fact, I think directly throw FileNotFoundException in 206 is better than IOException.
I have updated this part of the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171860521
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915462960
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> It is useful when the user modifies the contents of the file and reruns the task.
when the user modifies the contents of the file,what is the download param overwrite value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915527462
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using `if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) { return false; }`
The download method of interface StorageOperate has no return value. The return value of org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal is not processed in download. Are you sure you want to return a boolean value in S3 download?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS merged pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS merged PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171858983
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175686599
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [3 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [57.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [1.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915379783
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915460096
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
It is useful when the user modifies the contents of the file and reruns the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911560856
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
Review Comment:
> Can we directly use `org.apache.commons.io.Files.mkdirs(dstFileParentDirectory)`
I have updated this part of the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170089203
> > I found this PR dulicated with #10676. @sketchmind @guodongym
>
> @SbloodyS The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @sketchmind is willing to continue processing, I can scrap mine and change to Reviewer
As a first time contributor in ds, I is willing to continue,but review your code, why retrun instead of throw exception if file exists when overwrite is false. Usually this mode is append, failed or overwrite in other projects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170687317
I updated the code, if there are no new comments, I will keep it like this for now @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915391819
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
> > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
> >
> >
> > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
>
> You only need to `overwrite` this part to be consistent. The rest can remain the same.
Wait a minute, let me have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915423699
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
Review Comment:
We can use `Files.delete` instead of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915591907
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
What i mean is if there is no exception in `dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()`. We should always download s3 files to the local path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915391238
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
> Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
The copyHdfsToLocal indirectly causes an error when the dstFile path is a directory, but our code deletes it to avoid this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915529294
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using `if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) { return false; }`
If the parameter is always true, then it will not cause rerun error, should we keep our processing logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1196772625
Thanks for your contribution. This is a good beginning. Considering that it's your first contribution, I think we can get deep communication, you can contact me through mail or add WeChat(SbloodyS), when mail or added, please tell me who you are. I think I can help to familiar with the Dolphinscheduler if you meet with problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911129734
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
Review Comment:
* If the failed to create dir exception is not thrown in time, it will unnecessarily request s3 resources
* I will replaced it with FileUtils on next push
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911562243
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
Review Comment:
Please using standard way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911165698
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
Review Comment:
Ok, `mkdirs` will throw exception if createDir failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170065750
> I found this PR dulicated with #10676. @sketchmind @guodongym
I took another PR as mine and at first I was confused why it was different from my code until I found out it wasn't mine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915434039
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
Review Comment:
> We can use `Files.delete` instead of this.
ok,it has already been updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177078978
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> We should remove this return.
If so, It will result in unnecessary requests for s3 resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177081127
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] guodongym commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
guodongym commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170073148
> I found this PR dulicated with #10676. @sketchmind @guodongym
@SbloodyS The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @Sketchmind is willing to continue processing, I can scrap mine and change to Reviewer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170045203
I found this PR dulicated with #10676. @sketchmind @guodongym
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170086458
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914334304
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
Review Comment:
I think we should delete it instead of throwing exception if `overwrite` is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175682623
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10675](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c3b8424) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/3fbfa27f1343491ac7757cde59e4e5654a18e958?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3fbfa27) will **decrease** coverage by `0.07%`.
> The diff coverage is `34.13%`.
> :exclamation: Current head c3b8424 differs from pull request most recent head 7d7d522. Consider uploading reports for the commit 7d7d522 to get more accurate results
```diff
@@ Coverage Diff @@
## dev #10675 +/- ##
============================================
- Coverage 40.90% 40.82% -0.08%
+ Complexity 4830 4825 -5
============================================
Files 897 897
Lines 36157 36134 -23
Branches 4006 3990 -16
============================================
- Hits 14789 14753 -36
- Misses 19897 19915 +18
+ Partials 1471 1466 -5
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/api/service/impl/ProcessInstanceServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Qcm9jZXNzSW5zdGFuY2VTZXJ2aWNlSW1wbC5qYXZh) | `60.10% <ø> (ø)` | |
| [.../org/apache/dolphinscheduler/common/Constants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL0NvbnN0YW50cy5qYXZh) | `81.81% <ø> (ø)` | |
| [.../apache/dolphinscheduler/common/utils/S3Utils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL1MzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...er/server/builder/TaskExecutionContextBuilder.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL2J1aWxkZXIvVGFza0V4ZWN1dGlvbkNvbnRleHRCdWlsZGVyLmphdmE=) | `64.70% <0.00%> (-5.51%)` | :arrow_down: |
| [...r/server/master/runner/MasterSchedulerService.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvTWFzdGVyU2NoZWR1bGVyU2VydmljZS5qYXZh) | `0.00% <ø> (ø)` | |
| [...pand/TimePlaceholderResolverExpandServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvZXhwYW5kL1RpbWVQbGFjZWhvbGRlclJlc29sdmVyRXhwYW5kU2VydmljZUltcGwuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...nscheduler/service/process/ProcessServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvcHJvY2Vzcy9Qcm9jZXNzU2VydmljZUltcGwuamF2YQ==) | `30.28% <ø> (ø)` | |
| [...inscheduler/plugin/task/api/parser/ParamUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9wYXJzZXIvUGFyYW1VdGlscy5qYXZh) | `0.00% <ø> (ø)` | |
| [...heduler/plugin/task/api/parser/ParameterUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9wYXJzZXIvUGFyYW1ldGVyVXRpbHMuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...lphinscheduler/plugin/task/dq/DataQualityTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZGF0YXF1YWxpdHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svZHEvRGF0YVF1YWxpdHlUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3fbfa27...7d7d522](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177277634
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915421819
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
Review Comment:
>
If so, I would import a conflicting class with org.apache.dolphinscheduler.common.utils.FileUtils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r910744602
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
+ }
S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
try (S3ObjectInputStream s3is = o.getObjectContent();
- FileOutputStream fos = new FileOutputStream(dstFile)) {
+ FileOutputStream fos = new FileOutputStream(dstFilePath)) {
byte[] readBuf = new byte[1024];
- int readLen = 0;
+ int readLen;
while ((readLen = s3is.read(readBuf)) > 0) {
fos.write(readBuf, 0, readLen);
}
} catch (AmazonServiceException e) {
- logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+ logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
throw new IOException(e.getMessage());
} catch (FileNotFoundException e) {
- logger.error("the file isn`t exists");
- throw new IOException("the file isn`t exists");
+ logger.error("the destination file {} isn`t exists", dstFilePath);
+ throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);
Review Comment:
Please log the stack of e, or directly throw it. In fact, I think directly throw FileNotFoundException is better than IOException.
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ File dstFileParentDirectory = dstFile.getParentFile();
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+ throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+ }
Review Comment:
Can we directly use `org.apache.commons.io.Files.mkdirs(dstFileParentDirectory)`
```suggestion
FileUtils.mkdirs(dstFileParentDirectory)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911568735
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+ throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+ } else if (!overwrite) {
+ logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
Review Comment:
> Please using standard way.
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] guodongym commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
guodongym commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171358469
> > > I found this PR dulicated with #10676. @sketchmind @guodongym
> >
> >
> > @SbloodyS The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @sketchmind is willing to continue processing, I can scrap mine and change to Reviewer
>
> As a first time contributor in ds, I is willing to continue,but review your code, why retrun instead of throw exception if file exists when overwrite is false. Usually this mode is append, failed or overwrite in other projects
If the file already exists and you do not select overwrite, the file is available and you do not need to throw an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914357389
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (dstFile.isDirectory()) {
+ logger.error("destination file must be a file, but {} is a folder", dstFilePath);
Review Comment:
> > I think we should delete it instead of throwing exception if `overwrite` is true.
>
> Ok, by the way ,do I need to throw an error if the path is a directory and the overwrite if false
There is no need to verify it if it's false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915434402
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
Review Comment:
> Same as L183.
It has been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915388247
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists() && !overwrite) {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+ }
Review Comment:
> Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915455473
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
We should remove this return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915418296
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
+ }
+ } else {
+ org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
Review Comment:
Same as L183.
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
Review Comment:
Please use FileUtils.deleteDirectory instead of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915615298
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
}
@Override
- public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+ public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+ File dstFile = new File(dstFilePath);
+ if (dstFile.exists()) {
+ if (overwrite) {
+ if (dstFile.isDirectory()) {
+ Files.delete(dstFile.toPath());
+ }
+ } else {
+ logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+ return;
Review Comment:
> What i mean is if there is no exception in `dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()`. We should always download s3 files to the local path.
I have dropped overwrite related processing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179907499
> > LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
>
> Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code
This should add a comment in the code to address the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org