You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/31 23:20:03 UTC
[GitHub] [iceberg] rdblue opened a new pull request, #5683: Core: Fix exception handling in BaseTaskWriter
rdblue opened a new pull request, #5683:
URL: https://github.com/apache/iceberg/pull/5683
This is an alternative fix for the Flink double close problem, #4168 and #5310.
The original solution modified `S3OutputStream` so that every call to `close` after a failure would re-throw the original exception. That violates the [contract for `close`](https://docs.oracle.com/javase/8/docs/api/java/io/Closeable.html#close--), which states:
> Closes this stream and releases any system resources associated with it. **If the stream is already closed then invoking this method has no effect.**
The original fix also did not address the underlying problem that the stream was closed twice and still emitting data files. This PR fixes double close cases in `BaseTaskWriter` by ensuring the writer that is closed is always set to null.
This PR also attempts to fix the problem of duplicate data in Flink by throwing an `IllegalStateException` if a call to `close` fails but `complete` is still called on the `BaseTaskWriter`. The write result from `complete` is the only way to emit data files, so this ensures that no data files will be emitted from a writer that failed while closing a file. This is a precaution because it isn't clear why failed Flink writers were still emitting data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5683: Core: Fix exception handling in BaseTaskWriter
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5683:
URL: https://github.com/apache/iceberg/pull/5683#discussion_r960100144
##########
core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java:
##########
@@ -109,6 +116,8 @@ public void delete(CharSequence path, long pos, T row) {
public List<DeleteFile> complete() throws IOException {
close();
+ Preconditions.checkState(failure != null, "Cannot return results from failed writer", failure);
Review Comment:
This implements the same logic as in `BaseTaskWriter`: if there was an exception while writing the file, don't allow `complete` to succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks merged pull request #5683: Core: Fix exception handling in BaseTaskWriter
Posted by GitBox <gi...@apache.org>.
danielcweeks merged PR #5683:
URL: https://github.com/apache/iceberg/pull/5683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5683: Core: Fix exception handling in BaseTaskWriter
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5683:
URL: https://github.com/apache/iceberg/pull/5683#discussion_r960099888
##########
core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java:
##########
@@ -119,8 +128,8 @@ public CharSequenceSet referencedDataFiles() {
@Override
public void close() throws IOException {
if (!closed) {
- flushDeletes();
this.closed = true;
+ flushDeletes();
Review Comment:
Fixes double close here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org