You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/29 13:31:10 UTC

[GitHub] [dolphinscheduler] sketchmind opened a new pull request, #10675: [Fix-10665] [S3] Fix s3 download method

sketchmind opened a new pull request, #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675

   <!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
   
   
   ## Purpose of the pull request
   
   <!--(For example: This pull request adds checkstyle plugin).-->
   fix s3 download error, which cause task failure
   
   fix #10665
   ## Brief change log
   * when overwrite param is true and dest file is exist, throw exception
   * make dirs for ancestor directories of the source file path if it isn't exists, if create failed, throw exception 
   * To describe the error more clearly, change "the file isn`t exists" to specific error message
   <!--*(for example:)*
     - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   ## Verify this pull request
   
   <!--*(Please pick either of the following options)*-->
   
   This pull request is code cleanup without any test coverage.
   
   
   <!--*(example:)*
     - *Added dolphinscheduler-dao tests for end-to-end.*
     - *Added CronUtilsTest to verify the change.*
     - *Manually verified the change by testing locally.* -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1176959700

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [20 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![32.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25-16px.png '32.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.6% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915390535


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   > 
   > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
   
   You only need to `overwrite` this part to be consistent. The rest can remain the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > We should remove this return.
   
   If so, It will result in false of overwrite not being processed essentially



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911569143


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {

Review Comment:
   Looks like you've solved it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911568551


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {

Review Comment:
   > Please format this line.
   
   like this?
   ![image](https://user-images.githubusercontent.com/39169452/176811747-44931d40-9337-44cd-9eea-1564d721e7ff.png)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r910744602


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }
+        }
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {
             byte[] readBuf = new byte[1024];
-            int readLen = 0;
+            int readLen;
             while ((readLen = s3is.read(readBuf)) > 0) {
                 fos.write(readBuf, 0, readLen);
             }
         } catch (AmazonServiceException e) {
-            logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+            logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
             throw new IOException(e.getMessage());
         } catch (FileNotFoundException e) {
-            logger.error("the file isn`t exists");
-            throw new IOException("the file isn`t exists");
+            logger.error("the destination file {} isn`t exists", dstFilePath);
+            throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);

Review Comment:
   Please log the stack of e in line 203 or directly throw it. And in fact, I think directly throw FileNotFoundException in 206 is better than IOException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911561303


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {

Review Comment:
   Please format this line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1176960475

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [20 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![32.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25-16px.png '32.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.6% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915463868


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > It is useful when the user modifies the contents of the file and reruns the task.
   
   
   Do you mean that the destination file should be overwritten no matter what the overwrite parameter is?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > We should remove this return.
   
   If so, It will result in false of overwrite not being processed on the market cap



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179994987

   > > > LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
   > > 
   > > 
   > > Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code
   > 
   > This should add a comment in the code to address the reason.
   @SbloodyS Could you give me the code location of the rerun, I'll see if it has any effect
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1196783479

   > Thanks for your contribution. This is a good beginning. Considering that it's your first contribution, I think we can get deep communication, you can contact me through mail or add WeChat(SbloodyS), when mail or added, please tell me who you are. I think I can help to familiar with the Dolphinscheduler if you meet with problems.
   
   
   Thank you for your excellent suggestions and help during the submission process, hope the community is getting better and better


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170089919

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179891348

   > LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
   
   Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS  suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175686382

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [3 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![57.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/50-16px.png '57.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [57.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![1.3%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '1.3%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [1.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915418399


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   > > > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   > > > 
   > > > 
   > > > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
   > > 
   > > 
   > > You only need to `overwrite` this part to be consistent. The rest can remain the same.
   > 
   > Wait a minute, let me have a look
   
   
   
   > > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   > > 
   > > 
   > > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
   > 
   > You only need to `overwrite` this part to be consistent. The rest can remain the same.
   
   
   The code has been updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911563198


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {

Review Comment:
   > Please format this line.
   
   The code is already the same as the latest one you gave



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177278414

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915487719


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using
   `
   if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) {
       return false;
   }
   `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914356488


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);

Review Comment:
   > I think we should delete it instead of throwing exception if `overwrite` is true.
   
   Ok, by the way ,do I need to throw an error if the path is a directory and the overwrite if false



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911560936


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }
+        }
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {
             byte[] readBuf = new byte[1024];
-            int readLen = 0;
+            int readLen;
             while ((readLen = s3is.read(readBuf)) > 0) {
                 fos.write(readBuf, 0, readLen);
             }
         } catch (AmazonServiceException e) {
-            logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+            logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
             throw new IOException(e.getMessage());
         } catch (FileNotFoundException e) {
-            logger.error("the file isn`t exists");
-            throw new IOException("the file isn`t exists");
+            logger.error("the destination file {} isn`t exists", dstFilePath);
+            throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);

Review Comment:
   > Please log the stack of e in line 203 or directly throw it. And in fact, I think directly throw FileNotFoundException in 206 is better than IOException.
   
   I have updated this part of the code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171860521

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915462960


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > It is useful when the user modifies the contents of the file and reruns the task.
   when the user modifies the contents of the file,what is the download param overwrite value
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915527462


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using `if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) { return false; }`
   
   The download method of interface StorageOperate has no return value. The return value of org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal is not processed in download. Are you sure you want to return a boolean value in S3 download?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS merged pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS merged PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171858983

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175686599

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [3 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![57.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/50-16px.png '57.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [57.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![1.3%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '1.3%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [1.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915379783


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915460096


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   It is useful when the user modifies the contents of the file and reruns the task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911560856


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }

Review Comment:
   > Can we directly use `org.apache.commons.io.Files.mkdirs(dstFileParentDirectory)`
   
   I have updated this part of the code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170089203

   > > I found this PR dulicated with #10676. @sketchmind @guodongym
   > 
   > @SbloodyS The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @sketchmind is willing to continue processing, I can scrap mine and change to Reviewer
   
   As a first time contributor in ds, I is willing to continue,but  review your code,  why retrun instead of throw exception if file exists when overwrite is false. Usually this mode is append, failed or overwrite in other projects


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170687317

   I updated the code, if there are no new comments, I will keep it like this for now @SbloodyS 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915391819


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   > > > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   > > 
   > > 
   > > The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.
   > 
   > You only need to `overwrite` this part to be consistent. The rest can remain the same.
   
   
   Wait a minute, let me have a look



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915423699


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);

Review Comment:
   We can use `Files.delete` instead of this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915591907


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   What i mean is if there is no exception in `dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()`. We should always download s3 files to the local path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915391238


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   
   The copyHdfsToLocal indirectly causes an error when the dstFile path is a directory, but our code deletes it to avoid this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915529294


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > Currently `overwrite` is always true. I'm not sure why there is an overwrite parameter at present. And i think we should do this like `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal` using `if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) { return false; }`
   
   
   If the parameter is always true, then it will not cause rerun error, should we keep our processing logic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1196772625

   Thanks for your contribution. This is a good beginning. Considering that it's your first contribution, I think we can get deep communication, you can contact me through mail or add WeChat(SbloodyS), when mail or added, please tell me who you are. I think I can help to familiar with the Dolphinscheduler if you meet with problems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911129734


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }

Review Comment:
   * If the failed to create dir exception is not thrown in time, it will unnecessarily request s3 resources
   * I will replaced it with FileUtils on next push



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911562243


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);

Review Comment:
   Please using standard way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911165698


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }

Review Comment:
   Ok, `mkdirs` will throw exception if createDir failed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170065750

   > I found this PR dulicated with #10676. @sketchmind @guodongym
   
   I took another PR as mine and at first I was confused why it was different from my code until I found out it wasn't mine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915434039


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);

Review Comment:
   > We can use `Files.delete` instead of this.
   
   ok,it has already been updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177078978

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915458896


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > We should remove this return.
   
   If so, It will result in unnecessary requests for s3 resources



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177081127

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] guodongym commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
guodongym commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170073148

   > I found this PR dulicated with #10676. @sketchmind @guodongym
   
   @SbloodyS  The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @Sketchmind is willing to continue processing, I can scrap mine and change to Reviewer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170045203

   I found this PR dulicated with #10676. @sketchmind @guodongym 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1170086458

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914334304


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);

Review Comment:
   I think we should delete it instead of throwing exception if `overwrite` is true.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1175682623

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10675](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c3b8424) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/3fbfa27f1343491ac7757cde59e4e5654a18e958?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3fbfa27) will **decrease** coverage by `0.07%`.
   > The diff coverage is `34.13%`.
   
   > :exclamation: Current head c3b8424 differs from pull request most recent head 7d7d522. Consider uploading reports for the commit 7d7d522 to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #10675      +/-   ##
   ============================================
   - Coverage     40.90%   40.82%   -0.08%     
   + Complexity     4830     4825       -5     
   ============================================
     Files           897      897              
     Lines         36157    36134      -23     
     Branches       4006     3990      -16     
   ============================================
   - Hits          14789    14753      -36     
   - Misses        19897    19915      +18     
   + Partials       1471     1466       -5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...r/api/service/impl/ProcessInstanceServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Qcm9jZXNzSW5zdGFuY2VTZXJ2aWNlSW1wbC5qYXZh) | `60.10% <ø> (ø)` | |
   | [.../org/apache/dolphinscheduler/common/Constants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL0NvbnN0YW50cy5qYXZh) | `81.81% <ø> (ø)` | |
   | [.../apache/dolphinscheduler/common/utils/S3Utils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL1MzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...er/server/builder/TaskExecutionContextBuilder.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL2J1aWxkZXIvVGFza0V4ZWN1dGlvbkNvbnRleHRCdWlsZGVyLmphdmE=) | `64.70% <0.00%> (-5.51%)` | :arrow_down: |
   | [...r/server/master/runner/MasterSchedulerService.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvTWFzdGVyU2NoZWR1bGVyU2VydmljZS5qYXZh) | `0.00% <ø> (ø)` | |
   | [...pand/TimePlaceholderResolverExpandServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvZXhwYW5kL1RpbWVQbGFjZWhvbGRlclJlc29sdmVyRXhwYW5kU2VydmljZUltcGwuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [...nscheduler/service/process/ProcessServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvcHJvY2Vzcy9Qcm9jZXNzU2VydmljZUltcGwuamF2YQ==) | `30.28% <ø> (ø)` | |
   | [...inscheduler/plugin/task/api/parser/ParamUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9wYXJzZXIvUGFyYW1VdGlscy5qYXZh) | `0.00% <ø> (ø)` | |
   | [...heduler/plugin/task/api/parser/ParameterUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9wYXJzZXIvUGFyYW1ldGVyVXRpbHMuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [...lphinscheduler/plugin/task/dq/DataQualityTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZGF0YXF1YWxpdHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svZHEvRGF0YVF1YWxpdHlUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | ... and [31 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10675/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3fbfa27...7d7d522](https://codecov.io/gh/apache/dolphinscheduler/pull/10675?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1177277634

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10675)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10675&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10675&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915421819


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);

Review Comment:
   > 
   
   If so, I would import a conflicting class with org.apache.dolphinscheduler.common.utils.FileUtils
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r910744602


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }
+        }
         S3Object o = s3Client.getObject(BUCKET_NAME, srcFilePath);
         try (S3ObjectInputStream s3is = o.getObjectContent();
-             FileOutputStream fos = new FileOutputStream(dstFile)) {
+             FileOutputStream fos = new FileOutputStream(dstFilePath)) {
             byte[] readBuf = new byte[1024];
-            int readLen = 0;
+            int readLen;
             while ((readLen = s3is.read(readBuf)) > 0) {
                 fos.write(readBuf, 0, readLen);
             }
         } catch (AmazonServiceException e) {
-            logger.error("the resource can`t be downloaded,the bucket is {},and the src is {}", tenantCode, srcFilePath);
+            logger.error("the resource can`t be downloaded, the bucket is {}, and the src is {}", BUCKET_NAME, srcFilePath);
             throw new IOException(e.getMessage());
         } catch (FileNotFoundException e) {
-            logger.error("the file isn`t exists");
-            throw new IOException("the file isn`t exists");
+            logger.error("the destination file {} isn`t exists", dstFilePath);
+            throw new IOException("the destination file: " + dstFilePath + " isn`t exists", e);

Review Comment:
   Please log the stack of e, or directly throw it. In fact, I think directly throw FileNotFoundException is better than IOException.



##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,36 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        File dstFileParentDirectory = dstFile.getParentFile();
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            if (!dstFileParentDirectory.mkdirs() && !dstFileParentDirectory.exists()) {
+                throw new IOException("failed to create parent directory of destination file, directory is " + dstFileParentDirectory.getAbsolutePath());
+            }

Review Comment:
   Can we directly use `org.apache.commons.io.Files.mkdirs(dstFileParentDirectory)`
   ```suggestion
               FileUtils.mkdirs(dstFileParentDirectory)
               }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r911568735


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);
+                throw new IOException("destination file must be a file, but " + dstFilePath +" is a folder");
+            } else if (!overwrite) {
+                logger.info("the destination file {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else org.apache.commons.io.FileUtils.createParentDirectories(dstFile);

Review Comment:
   > Please using standard way.
   
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] guodongym commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
guodongym commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1171358469

   > > > I found this PR dulicated with #10676. @sketchmind @guodongym
   > > 
   > > 
   > > @SbloodyS The afternoon issue was assigned to me, so I developed it. After submit PR, I also found some duplication. If @sketchmind is willing to continue processing, I can scrap mine and change to Reviewer
   > 
   > As a first time contributor in ds, I is willing to continue,but review your code, why retrun instead of throw exception if file exists when overwrite is false. Usually this mode is append, failed or overwrite in other projects
   
   If the file already exists and you do not select overwrite, the file is available and you do not need to throw an exception


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r914357389


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,32 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (dstFile.isDirectory()) {
+                logger.error("destination file must be a file, but {} is a folder", dstFilePath);

Review Comment:
   > > I think we should delete it instead of throwing exception if `overwrite` is true.
   > 
   > Ok, by the way ,do I need to throw an error if the path is a directory and the overwrite if false
   
   There is no need to verify it if it's false.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915434402


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);

Review Comment:
   > Same as L183.
   
   It has been resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915388247


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,30 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists() && !overwrite) {
+            logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+            return;
+        }
+        if (dstFile.isDirectory()) {
+            org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);
+        }

Review Comment:
   > Could you keep this consistent with `org.apache.dolphinscheduler.common.utils.HadoopUtils#copyHdfsToLocal`?
   
   The processing logic on both sides is different, and the return value type of the function is also different. I suggest that another PR can be opened for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915455473


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   We should remove this return.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915418296


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;
+            }
+        } else {
+            org.apache.commons.io.FileUtils.createParentDirectories(dstFile);

Review Comment:
   Same as L183.



##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +175,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    org.apache.commons.io.FileUtils.deleteDirectory(dstFile);

Review Comment:
   Please use FileUtils.deleteDirectory instead of this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sketchmind commented on a diff in pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
sketchmind commented on code in PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#discussion_r915615298


##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java:
##########
@@ -175,21 +176,33 @@ public String getFileName(ResourceType resourceType, String tenantCode, String f
     }
 
     @Override
-    public void download(String tenantCode, String srcFilePath, String dstFile, boolean deleteSource, boolean overwrite) throws IOException {
+    public void download(String tenantCode, String srcFilePath, String dstFilePath, boolean deleteSource, boolean overwrite) throws IOException {
+        File dstFile = new File(dstFilePath);
+        if (dstFile.exists()) {
+            if (overwrite) {
+                if (dstFile.isDirectory()) {
+                    Files.delete(dstFile.toPath());
+                }
+            } else {
+                logger.info("the destination path {} already exists, download operation will be ignored", dstFilePath);
+                return;

Review Comment:
   > What i mean is if there is no exception in `dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()`. We should always download s3 files to the local path.
   
   I have dropped overwrite related processing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #10675: [Fix-10665] [S3] Fix s3 download method

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #10675:
URL: https://github.com/apache/dolphinscheduler/pull/10675#issuecomment-1179907499

   > > LGTM, nit: the `overwrite` is not be used? it's better to implements this, if we set overwrite then delete the exist dstfile, if we don't set overwrite we don't delete the exist dstFile event it is a directly.
   > 
   > Param overwrite is always true in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadResource and the existence of the file has been handled in advance in org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread#downloadCheck,I think the overriding related processing logic should be kept. @SbloodyS suggested to remove this part considering the situation of task rerun,I don't know if it will affect the rerun, so I removed this part of the code
   
   This should add a comment in the code to address the reason.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org