You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/16 00:29:13 UTC
[GitHub] [beam] pabloem opened a new pull request #13558: FileIO stops overwriting files on retries
pabloem opened a new pull request #13558:
URL: https://github.com/apache/beam/pull/13558
In this case, the default behavior for FileIO changes slightly:
- Before this change, FileIO copies temp files into destination locations without checking whether destinations exist.
- With the new change, if the destination file exists already, nothing will be copied.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780855078
I don't think HDFS provides retention policies (but imagine e.g. an S3 bucket with retention policy being accessed via HDFS would use HadoopFilesystem and have a retention policy)
I might consider the question differently:
- If the checksum for the file is equal, it is very, very likely that the file is the same.
- If the filesystem does not provide a checksum for the file, then it is not possible to know if it's the same file - even if their sizes are equal.
So what I would propose we do differently instead is:
- If the files have equal checksum, we skip rewriting them
- If the filesystem does not provide a checksum, then we will always overwrite (instead of matching on file size)
This effectively means that for HDFS, we will always overwrite, which is what you propose - but it's not related to whether the FS supports a retention policy, but rather to whether we can be confident that the file contents are the same or not. Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780888111
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780750912
run dataflow validatesrunnner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577215841
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = matchResources(destResourceIds);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747623935
Added a JIRA
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781100965
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577357032
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
##########
@@ -764,7 +764,11 @@ final void moveToOutputFiles(
}
// During a failure case, files may have been deleted in an earlier step. Thus
// we ignore missing files here.
- FileSystems.rename(srcFiles, dstFiles, StandardMoveOptions.IGNORE_MISSING_FILES);
+ FileSystems.rename(
+ srcFiles,
+ dstFiles,
+ StandardMoveOptions.IGNORE_MISSING_FILES,
+ StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS);
Review comment:
We've verified that this is the correct behavior for GCS. What about other file systems?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,16 +412,40 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = new ArrayList<>();
+ if (skipExistingDest) {
+ matchDestResults = matchResources(destResourceIds);
+ }
+
+ for (int i = 0; i < matchSrcResults.size(); ++i) {
+ if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+ // If the source is not found, and we are ignoring found source files, then we skip it.
+ continue;
}
+ if (skipExistingDest
+ && matchDestResults.get(i).status().equals(Status.OK)
+ && filesMatch(
+ matchDestResults.get(i).metadata().get(0),
+ matchSrcResults.get(i).metadata().get(0))) {
+ // If the destination exists, and we are skipping when destinations exist, then we skip.
+ continue;
+ }
+ srcToHandle.add(srcResourceIds.get(i));
+ destToHandle.add(destResourceIds.get(i));
}
return KV.of(srcToHandle, destToHandle);
}
+ private static boolean filesMatch(MatchResult.Metadata first, MatchResult.Metadata second) {
+ if (!first.checksum().isPresent() && !second.checksum().isPresent()) {
Review comment:
Should this be ||?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780755111
Just to clarify, this avoid overwriting "old" files that pre-dated the pipeline run, does it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780752000
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-767758056
Considering these results, then we can conclude:
1. If the file appears when listing files in a bucket/directory, it means that it has been written successfully
2. If a bundle fails in the middle of writing a file (and the file was not finished), this file will not show up in directory listing
- Furthermore, the file can be safely retried - a new `rewriteToken` will be created for it
- The bucket-lock policy will not prevent us from starting the `rewrite` for this file again
So I think this change is safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577810302
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
##########
@@ -764,7 +764,11 @@ final void moveToOutputFiles(
}
// During a failure case, files may have been deleted in an earlier step. Thus
// we ignore missing files here.
- FileSystems.rename(srcFiles, dstFiles, StandardMoveOptions.IGNORE_MISSING_FILES);
+ FileSystems.rename(
+ srcFiles,
+ dstFiles,
+ StandardMoveOptions.IGNORE_MISSING_FILES,
+ StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS);
Review comment:
the behavior that we checked in GCS is that we will not encounter 'incomplete' files. But in fact, we only consider a file 'incomplete' if it has a different checksum (or size in absence of checksum). For other file systems, the same rationale applies: Encountering a file with equal checksum means we have the exact same file, and we don't need to rewrite it.
Only filesystem where this is a tough assumption is HadoopFileSystem, where we don't have a hash function, and instead we rely solely on the size.
Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577216075
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+ for (int i = 0; i < matchSrcResults.size(); ++i) {
+ if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+ // If the source is not found, and we are ignoring found source files, then we skip it.
+ continue;
+ }
+ if (matchDestResults.get(i).status().equals(Status.OK)
+ && matchDestResults.get(i).metadata().get(0).sizeBytes()
+ == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+ && skipExistingDest) {
Review comment:
I have added this for GCS in the latest commit, but defaulting to size for other filesystems. LMK what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780200714
oh well, I also updated other filesystems after all
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780726161
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780890411
Run Java ManyFilesTextIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780759786
No. This is to avoid overwriting files that were successfully written in a previous bundle execution, but the bundle is being retried due to a later failure.
We do this because some users are reporting issues where pipelines experience errors on retries for GCS buckets with specific retention policies (files cannot be deleted, and thus cannot be overwritten)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-767757877
Hi Reuven - I ran some experiments on various GCS buckets. Here's what I found
The experiments were done like so:
- A file of 1.6 MB was copied with `rewrite`
- A total of 1MB was copied on every call to `rewrite` (this is configurable in the call)
- Source bucket: My bucket, without any special policies.
**Experiment 1** - Normal, successful workflow
- Destination bucket: Bucket in different region. No retention policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
**Experiment 2** - Call has to be retried
- Destination bucket: Bucket in different region. No retention policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
- After running first `rewrite` call again
- Call is successful
- Response contains a **new `rewriteToken`**
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
**Experiment 3** - Call has to be retried - bucket has retention policy
- Destination bucket: Bucket in different region. Bucket has retention policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
- After running first `rewrite` call again
- Call is successful
- Response contains a **new `rewriteToken`**
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls gs://bucket/file`
- Trying to delete file
- Call fails. Unable to delete file.
- Trying to overwrite file with new `rewrite` call
- Call fails. Unable to delete file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780888280
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780188865
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781079229
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780811179
Do other file systems have these retention policies? If not, is it safe to
just continue overwriting in the HDFS case?
On Wed, Feb 17, 2021 at 10:32 AM Pablo <no...@github.com> wrote:
> No. This is to avoid overwriting files that were successfully written
> previously, but are part of a bundle that is being retried.
>
> We do this because some users are reporting issues where pipelines
> experience errors on retries for GCS buckets with specific retention
> policies (files cannot be deleted, and thus cannot be overwritten)
>
> —
> You are receiving this because your review was requested.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/13558#issuecomment-780759786>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVIHTFPPZXPZ4BN6TJ3S7QDSRANCNFSM4U5FVBKA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem removed a comment on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem removed a comment on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-781079229
Please give a final LGTM so I can merge : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780860713
That sounds like a good plan to me.
On Wed, Feb 17, 2021 at 1:10 PM Pablo <no...@github.com> wrote:
> I don't think HDFS provides retention policies (but imagine e.g. an S3
> bucket with retention policy being accessed via HDFS would use
> HadoopFilesystem and have a retention policy)
>
> I might consider the question differently:
>
> - If the checksum for the file is equal, it is very, very likely that
> the file is the same.
> - If the filesystem does not provide a checksum for the file, then it
> is not possible to know if it's the same file - even if their sizes are
> equal.
>
> So what I would propose we do differently instead is:
>
> - If the files have equal checksum, we skip rewriting them
> - If the filesystem does not provide a checksum, then we will always
> overwrite (instead of matching on file size)
>
> This effectively means that for HDFS, we will always overwrite, which is
> what you propose - but it's not related to whether the FS supports a
> retention policy, but rather to whether we can be confident that the file
> contents are the same or not. Thoughts?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/13558#issuecomment-780855078>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADWVANVP2QAD4YF5RM4A3DS7QWDBANCNFSM4U5FVBKA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780759786
No. This is to avoid overwriting files that were successfully written previously, but are part of a bundle that is being retried.
We do this because some users are reporting issues where pipelines experience errors on retries for GCS buckets with specific retention policies (files cannot be deleted, and thus cannot be overwritten)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747196442
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577368164
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,16 +412,40 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = new ArrayList<>();
+ if (skipExistingDest) {
+ matchDestResults = matchResources(destResourceIds);
+ }
+
+ for (int i = 0; i < matchSrcResults.size(); ++i) {
+ if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+ // If the source is not found, and we are ignoring found source files, then we skip it.
+ continue;
}
+ if (skipExistingDest
+ && matchDestResults.get(i).status().equals(Status.OK)
+ && filesMatch(
+ matchDestResults.get(i).metadata().get(0),
+ matchSrcResults.get(i).metadata().get(0))) {
+ // If the destination exists, and we are skipping when destinations exist, then we skip.
+ continue;
+ }
+ srcToHandle.add(srcResourceIds.get(i));
+ destToHandle.add(destResourceIds.get(i));
}
return KV.of(srcToHandle, destToHandle);
}
+ private static boolean filesMatch(MatchResult.Metadata first, MatchResult.Metadata second) {
+ if (!first.checksum().isPresent() && !second.checksum().isPresent()) {
Review comment:
changed this to null. If both checksums are nullable, only then should we rely on the file size - otherwise we should always rely on the checksum (if only one file reports a checksum and the other doesnt, then they are not equal, which is what happens in the next section)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on a change in pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13558:
URL: https://github.com/apache/beam/pull/13558#discussion_r577167815
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = matchResources(destResourceIds);
Review comment:
only populate matchDestResults if skipExistingDest is ste.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+ for (int i = 0; i < matchSrcResults.size(); ++i) {
+ if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+ // If the source is not found, and we are ignoring found source files, then we skip it.
+ continue;
+ }
+ if (matchDestResults.get(i).status().equals(Status.OK)
+ && matchDestResults.get(i).metadata().get(0).sizeBytes()
+ == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+ && skipExistingDest) {
Review comment:
put skipExistingDest first in the conditional.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
##########
@@ -401,12 +412,23 @@ public ResourceId apply(@Nonnull Metadata input) {
List<ResourceId> srcToHandle = new ArrayList<>();
List<ResourceId> destToHandle = new ArrayList<>();
- List<MatchResult> matchResults = matchResources(srcResourceIds);
- for (int i = 0; i < matchResults.size(); ++i) {
- if (!matchResults.get(i).status().equals(Status.NOT_FOUND)) {
- srcToHandle.add(srcResourceIds.get(i));
- destToHandle.add(destResourceIds.get(i));
+ List<MatchResult> matchSrcResults = matchResources(srcResourceIds);
+ List<MatchResult> matchDestResults = matchResources(destResourceIds);
+
+ for (int i = 0; i < matchSrcResults.size(); ++i) {
+ if (matchSrcResults.get(i).status().equals(Status.NOT_FOUND) && ignoreMissingSrc) {
+ // If the source is not found, and we are ignoring found source files, then we skip it.
+ continue;
+ }
+ if (matchDestResults.get(i).status().equals(Status.OK)
+ && matchDestResults.get(i).metadata().get(0).sizeBytes()
+ == matchSrcResults.get(i).metadata().get(0).sizeBytes()
+ && skipExistingDest) {
Review comment:
Wish there was something better than byte size we could use here (e.g. file hash).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13558: FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-745683317
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780751531
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: [BEAM-11494] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780860594
sgtm
On Wed, Feb 17, 2021 at 1:10 PM Pablo <no...@github.com> wrote:
> I don't think HDFS provides retention policies (but imagine e.g. an S3
> bucket with retention policy being accessed via HDFS would use
> HadoopFilesystem and have a retention policy)
>
> I might consider the question differently:
>
> - If the checksum for the file is equal, it is very, very likely that
> the file is the same.
> - If the filesystem does not provide a checksum for the file, then it
> is not possible to know if it's the same file - even if their sizes are
> equal.
>
> So what I would propose we do differently instead is:
>
> - If the files have equal checksum, we skip rewriting them
> - If the filesystem does not provide a checksum, then we will always
> overwrite (instead of matching on file size)
>
> This effectively means that for HDFS, we will always overwrite, which is
> what you propose - but it's not related to whether the FS supports a
> retention policy, but rather to whether we can be confident that the file
> contents are the same or not. Thoughts?
>
> —
> You are receiving this because your review was requested.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/13558#issuecomment-780855078>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVKGM7ODFZUPUJQHLGTS7QWDBANCNFSM4U5FVBKA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #13558: FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-747208015
Is there a JIRA that explains the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #13558: [BEAM-11494][BEAM-11821] FileIO stops overwriting files on retries
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #13558:
URL: https://github.com/apache/beam/pull/13558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org