You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/16 00:29:13 UTC

[GitHub] [beam] pabloem opened a new pull request #13558: FileIO stops overwriting files on retries

pabloem opened a new pull request #13558:
URL: https://github.com/apache/beam/pull/13558


   In this case, the default behavior for FileIO changes slightly: 
   - Before this change, FileIO copies temp files into destination locations without checking whether destinations exist. 
   - With the new change, if the destination file exists already, nothing will be copied.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam
 .apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.a
 pache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780855078


   I don't think HDFS provides retention policies (but imagine e.g. an S3 bucket with retention policy being accessed via HDFS would use HadoopFilesystem and have a retention policy)
   
   I might consider the question differently:
   - If the checksum for the file is equal, it is very, very likely that the file is the same.
   - If the filesystem does not provide a checksum for the file, then it is not possible to know if it's the same file - even if their sizes are equal.
   
   So what I would propose we do differently instead is:
   - If the files have equal checksum, we skip rewriting them
   - If the filesystem does not provide a checksum, then we will always overwrite (instead of matching on file size)
   
   This effectively means that for HDFS, we will always overwrite, which is what you propose - but it's not related to whether the FS supports a retention policy, but rather to whether we can be confident that the file contents are the same or not. Thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780888111






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780750912


   run dataflow validatesrunnner


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577215841



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = matchResources(destResourceIds);

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747623935


   Added a JIRA


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781100965


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577357032



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
##########
@@ -764,7 +764,11 @@ final void moveToOutputFiles(
       }
       // During a failure case, files may have been deleted in an earlier step. Thus
       // we ignore missing files here.
-      FileSystems.rename(srcFiles, dstFiles, StandardMoveOptions.IGNORE_MISSING_FILES);
+      FileSystems.rename(
+          srcFiles,
+          dstFiles,
+          StandardMoveOptions.IGNORE_MISSING_FILES,
+          StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS);

Review comment:
       We've verified that this is the correct behavior for GCS. What about other file systems?

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,16 +412,40 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = new ArrayList<>();
+    if (skipExistingDest) {
+      matchDestResults = matchResources(destResourceIds);
+    }
+
+    for (int i = 0; i < matchSrcResults.size(); ++i) {
+      if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+        // If the source is not found, and we are ignoring found source files, then we skip it.
+        continue;
       }
+      if (skipExistingDest
+          && matchDestResults.get(i).status().equals(Status.OK)
+          && filesMatch(
+              matchDestResults.get(i).metadata().get(0),
+              matchSrcResults.get(i).metadata().get(0))) {
+        // If the destination exists, and we are skipping when destinations exist, then we skip.
+        continue;
+      }
+      srcToHandle.add(srcResourceIds.get(i));
+      destToHandle.add(destResourceIds.get(i));
     }
     return KV.of(srcToHandle, destToHandle);
   }
 
+  private static boolean filesMatch(MatchResult.Metadata first, MatchResult.Metadata second) {
+    if (!first.checksum().isPresent() && !second.checksum().isPresent()) {

Review comment:
       Should this be ||?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] robertwb commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780755111


   Just to clarify, this avoid overwriting "old" files that pre-dated the pipeline run, does it? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780752000


   Run Spark ValidatesRunner


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-767758056


   Considering these results, then we can conclude:
   
   1. If the file appears when listing files in a bucket/directory, it means that it has been written successfully
   2. If a bundle fails in the middle of writing a file (and the file was not finished), this file will not show up in directory listing
      - Furthermore, the file can be safely retried - a new `rewriteToken` will be created for it
      - The bucket-lock policy will not prevent us from starting the `rewrite` for this file again
   
   So I think this change is safe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577810302



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
##########
@@ -764,7 +764,11 @@ final void moveToOutputFiles(
       }
       // During a failure case, files may have been deleted in an earlier step. Thus
       // we ignore missing files here.
-      FileSystems.rename(srcFiles, dstFiles, StandardMoveOptions.IGNORE_MISSING_FILES);
+      FileSystems.rename(
+          srcFiles,
+          dstFiles,
+          StandardMoveOptions.IGNORE_MISSING_FILES,
+          StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS);

Review comment:
       the behavior that we checked in GCS is that we will not encounter 'incomplete' files. But in fact, we only consider a file 'incomplete' if it has a different checksum (or size in absence of checksum). For other file systems, the same rationale applies: Encountering a file with equal checksum means we have the exact same file, and we don't need to rewrite it.
   
   Only filesystem where this is a tough assumption is HadoopFileSystem, where we don't have a hash function, and instead we rely solely on the size.
   
   Thoughts?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577216075



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+    for (int i = 0; i < matchSrcResults.size(); ++i) {
+      if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+        // If the source is not found, and we are ignoring found source files, then we skip it.
+        continue;
+      }
+      if (matchDestResults.get(i).status().equals(Status.OK)
+          && matchDestResults.get(i).metadata().get(0).sizeBytes()
+              == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+          && skipExistingDest) {

Review comment:
       I have added this for GCS in the latest commit, but defaulting to size for other filesystems. LMK what you think.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780200714


   oh well, I also updated other filesystems after all


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780726161


   Run Java PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780890411


   Run Java ManyFilesTextIO Performance Test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem edited a comment on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780759786


   No. This is to avoid overwriting files that were successfully written in a previous bundle execution, but the bundle is being retried due to a later failure.
   
   We do this because some users are reporting issues where pipelines experience errors on retries for GCS buckets with specific retention policies (files cannot be deleted, and thus cannot be overwritten)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-767757877


   Hi Reuven - I ran some experiments on various GCS buckets. Here's what I found
   
   The experiments were done like so:
   - A file of 1.6 MB was copied with `rewrite`
   - A total of 1MB was copied on every call to `rewrite` (this is configurable in the call)
   - Source bucket: My bucket, without any special policies.
   
   **Experiment 1** - Normal, successful workflow
   - Destination bucket: Bucket in different region. No retention policy.
   - After first `rewrite` call
     - Call is successful
     - Response contains a `rewriteToken`
     - `done: False` in response
     - File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - After second `rewrite` call
     - Call is successful
     - `done: True` in response
     - File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
   
   **Experiment 2** - Call has to be retried
   - Destination bucket: Bucket in different region. No retention policy.
   - After first `rewrite` call
     - Call is successful
     - Response contains a `rewriteToken`
     - `done: False` in response
     - File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - After running first `rewrite` call again
     - Call is successful
     - Response contains a **new `rewriteToken`**
     - `done: False` in response
     - File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - After second `rewrite` call
     - Call is successful
     - `done: True` in response
     - File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
   
   **Experiment 3** - Call has to be retried - bucket has retention policy
   - Destination bucket: Bucket in different region. Bucket has retention policy.
   - After first `rewrite` call
     - Call is successful
     - Response contains a `rewriteToken`
     - `done: False` in response
     - File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - After running first `rewrite` call again
     - Call is successful
     - Response contains a **new `rewriteToken`**
     - `done: False` in response
     - File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - After second `rewrite` call
     - Call is successful
     - `done: True` in response
     - File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
   - Trying to delete file
     - Call fails. Unable to delete file.
   - Trying to overwrite file with new `rewrite` call
     - Call fails. Unable to delete file.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780888280






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780188865


   Run Java PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781079229






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780811179


   Do other file systems have these retention policies? If not, is it safe to
   just continue overwriting in the HDFS case?
   
   On Wed, Feb 17, 2021 at 10:32 AM Pablo <no...@github.com> wrote:
   
   > No. This is to avoid overwriting files that were successfully written
   > previously, but are part of a bundle that is being retried.
   >
   > We do this because some users are reporting issues where pipelines
   > experience errors on retries for GCS buckets with specific retention
   > policies (files cannot be deleted, and thus cannot be overwritten)
   >
   > —
   > You are receiving this because your review was requested.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/13558#issuecomment-780759786>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AFAYJVIHTFPPZXPZ4BN6TJ3S7QDSRANCNFSM4U5FVBKA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem removed a comment on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem removed a comment on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781079229


   Please give a final LGTM so I can merge : ) 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] robertwb commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780860713


   That sounds like a good plan to me.
   
   On Wed, Feb 17, 2021 at 1:10 PM Pablo <no...@github.com> wrote:
   
   > I don't think HDFS provides retention policies (but imagine e.g. an S3
   > bucket with retention policy being accessed via HDFS would use
   > HadoopFilesystem and have a retention policy)
   >
   > I might consider the question differently:
   >
   >    - If the checksum for the file is equal, it is very, very likely that
   >    the file is the same.
   >    - If the filesystem does not provide a checksum for the file, then it
   >    is not possible to know if it's the same file - even if their sizes are
   >    equal.
   >
   > So what I would propose we do differently instead is:
   >
   >    - If the files have equal checksum, we skip rewriting them
   >    - If the filesystem does not provide a checksum, then we will always
   >    overwrite (instead of matching on file size)
   >
   > This effectively means that for HDFS, we will always overwrite, which is
   > what you propose - but it's not related to whether the FS supports a
   > retention policy, but rather to whether we can be confident that the file
   > contents are the same or not. Thoughts?
   >
   > —
   > You are receiving this because you commented.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/13558#issuecomment-780855078>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AADWVANVP2QAD4YF5RM4A3DS7QWDBANCNFSM4U5FVBKA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780759786


   No. This is to avoid overwriting files that were successfully written previously, but are part of a bundle that is being retried.
   
   We do this because some users are reporting issues where pipelines experience errors on retries for GCS buckets with specific retention policies (files cannot be deleted, and thus cannot be overwritten)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747196442


   Run Java PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577368164



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,16 +412,40 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = new ArrayList<>();
+    if (skipExistingDest) {
+      matchDestResults = matchResources(destResourceIds);
+    }
+
+    for (int i = 0; i < matchSrcResults.size(); ++i) {
+      if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+        // If the source is not found, and we are ignoring found source files, then we skip it.
+        continue;
       }
+      if (skipExistingDest
+          && matchDestResults.get(i).status().equals(Status.OK)
+          && filesMatch(
+              matchDestResults.get(i).metadata().get(0),
+              matchSrcResults.get(i).metadata().get(0))) {
+        // If the destination exists, and we are skipping when destinations exist, then we skip.
+        continue;
+      }
+      srcToHandle.add(srcResourceIds.get(i));
+      destToHandle.add(destResourceIds.get(i));
     }
     return KV.of(srcToHandle, destToHandle);
   }
 
+  private static boolean filesMatch(MatchResult.Metadata first, MatchResult.Metadata second) {
+    if (!first.checksum().isPresent() && !second.checksum().isPresent()) {

Review comment:
       changed this to null. If both checksums are nullable, only then should we rely on the file size - otherwise we should always rely on the checksum (if only one file reports a checksum and the other doesnt, then they are not equal, which is what happens in the next section)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577167815



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = matchResources(destResourceIds);

Review comment:
       only populate matchDestResults if skipExistingDest is ste.

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+    for (int i = 0; i < matchSrcResults.size(); ++i) {
+      if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+        // If the source is not found, and we are ignoring found source files, then we skip it.
+        continue;
+      }
+      if (matchDestResults.get(i).status().equals(Status.OK)
+          && matchDestResults.get(i).metadata().get(0).sizeBytes()
+              == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+          && skipExistingDest) {

Review comment:
       put skipExistingDest first in the conditional.

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
     List<ResourceId> srcToHandle = new ArrayList<>();
     List<ResourceId> destToHandle = new ArrayList<>();
 
-    List<MatchResult> matchResults = matchResources(srcResourceIds);
-    for (int i = 0; i < matchResults.size(); ++i) {
-      if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
-        srcToHandle.add(srcResourceIds.get(i));
-        destToHandle.add(destResourceIds.get(i));
+    List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+    List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+    for (int i = 0; i < matchSrcResults.size(); ++i) {
+      if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+        // If the source is not found, and we are ignoring found source files, then we skip it.
+        continue;
+      }
+      if (matchDestResults.get(i).status().equals(Status.OK)
+          && matchDestResults.get(i).metadata().get(0).sizeBytes()
+              == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+          && skipExistingDest) {

Review comment:
       Wish there was something better than byte size we could use here (e.g. file hash).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem commented on pull request #13558: FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-745683317


   Run Java PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780751531


   Run Dataflow ValidatesRunner


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780860594


   sgtm
   
   On Wed, Feb 17, 2021 at 1:10 PM Pablo <no...@github.com> wrote:
   
   > I don't think HDFS provides retention policies (but imagine e.g. an S3
   > bucket with retention policy being accessed via HDFS would use
   > HadoopFilesystem and have a retention policy)
   >
   > I might consider the question differently:
   >
   >    - If the checksum for the file is equal, it is very, very likely that
   >    the file is the same.
   >    - If the filesystem does not provide a checksum for the file, then it
   >    is not possible to know if it's the same file - even if their sizes are
   >    equal.
   >
   > So what I would propose we do differently instead is:
   >
   >    - If the files have equal checksum, we skip rewriting them
   >    - If the filesystem does not provide a checksum, then we will always
   >    overwrite (instead of matching on file size)
   >
   > This effectively means that for HDFS, we will always overwrite, which is
   > what you propose - but it's not related to whether the FS supports a
   > retention policy, but rather to whether we can be confident that the file
   > contents are the same or not. Thoughts?
   >
   > —
   > You are receiving this because your review was requested.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/13558#issuecomment-780855078>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AFAYJVKGM7ODFZUPUJQHLGTS7QWDBANCNFSM4U5FVBKA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13558: FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747208015


   Is there a JIRA that explains the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] pabloem merged pull request #13558: [BEAM-11494][BEAM-11821] FileIO stops overwriting files on retries

Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #13558:
URL: https://github.com/apache/beam/pull/13558


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org