You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fpetersen-gl (via GitHub)" <gi...@apache.org> on 2025/07/16 08:22:42 UTC
[PR] Closing the ParquetWriter in the correct spot [iceberg]
fpetersen-gl opened a new pull request, #13565:
URL: https://github.com/apache/iceberg/pull/13565
Closes #13508
This PR tries to avoid data loss when it comes to writing parquet-files over an unstable network.
In the ParquetWriter, the internal flag `closed` is set before all logic has been performed successfully. If a wrapped writer is later calling `close()` again, this writer doesn't do anything, as it considers itself closed already. This leads to a missing parquet-file, which is reported back as successfully written.
The accompanying test tries to mimic the behaviour of an unstable network being used by e.g. an S3-client by first keeping all data in memory before writing it to a file when `close()` is called on the output. Depending on a flag, this fails or succeeds. The various bits of the test have been copy-pasted from already existing test-code, as they can't be re-used due to their too strict encapsulation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3088852737
@ajantha-bhat or @nastra , could you please have another look at the test? It's now focused on the `ParquetWriter`, without all the wrapping classes.
Also, a ticket in parquet-java exists now, to hopefully fix it on that side soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3094340108
> @ajantha-bhat or @nastra , could you please have another look at the test? It's now focused on the ParquetWriter, without all the wrapping classes.
Also, a ticket in parquet-java exists now, to hopefully fix it on that side soon.
I checked the latest state of the PR. I am happy with the tests now (it is clean and compact as suggested). And I would agree with @nastra that we should merge this PR with/after parquet version bump as underlaying fix really needed to fix this issue properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3203354208
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209983354
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
+ }
+
+ protected Record toRow(Integer id, String data) {
+ GenericRecord result = GenericRecord.create(table.schema().asStruct());
+ result.set(0, id);
+ result.set(1, data);
+ return result;
+ }
+
+ private static class MyFileWriterFactory extends BaseFileWriterFactory<Record> {
+ protected MyFileWriterFactory(Table table) {
+ super(
+ table,
+ FileFormat.PARQUET,
+ table.schema(),
+ table.sortOrder(),
+ FileFormat.PARQUET,
+ null,
+ table.schema(),
+ table.sortOrder(),
+ table.schema());
+ }
+
+ @Override
+ protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
+ builder.createWriterFunc(GenericParquetWriter::create);
+ }
+
+ @Override
+ protected void configureDataWrite(Avro.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureDataWrite(ORC.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+ static final FileIO IO = new FailingLocalFileIo();
+
+ private static class FailingLocalFileIo extends TestTables.LocalFileIO {
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return new FailingLocalOutputFile(Paths.get(path).toAbsolutePath().toFile());
+ }
+ }
+
+ private static class FailingLocalOutputFile implements OutputFile {
+ private final File file;
+
+ private FailingLocalOutputFile(File file) {
+ this.file = file;
+ }
+
+ @Override
+ public PositionOutputStream create() {
+ if (file.exists()) {
+ throw new AlreadyExistsException("File already exists: %s", file);
+ }
+
+ if (!file.getParentFile().isDirectory() && !file.getParentFile().mkdirs()) {
+ throw new RuntimeIOException(
+ "Failed to create the file's directory at %s.", file.getParentFile().getAbsolutePath());
+ }
+
+ PARQUET_FILE_LIST.add(file);
+
+ return new PositionFileOutputStream(file);
+ }
+
+ @Override
+ public PositionOutputStream createOrOverwrite() {
+ if (file.exists() && !file.delete()) {
+ throw new RuntimeIOException("Failed to delete: %s", file);
+ }
+
+ return create();
+ }
+
+ @Override
+ public String location() {
+ return file.toString();
+ }
+
+ @Override
+ public InputFile toInputFile() {
+ return Files.localInput(file);
+ }
+
+ @Override
+ public String toString() {
+ return location();
+ }
+ }
+
+ /**
+ * Keeps all data in memory, before writing it to file when calling close().
+ *
+ * <p>This mimics the behaviour of writing to S3 over network - files are only uploaded when
+ * closing the output.
+ */
+ private static class PositionFileOutputStream extends PositionOutputStream {
+ private final ByteArrayOutputStream contents = new ByteArrayOutputStream();
+ private final File file;
+
+ private PositionFileOutputStream(File file) {
+ this.file = file;
+ }
+
+ @Override
+ public long getPos() {
+ return contents.size();
+ }
+
+ @Override
+ public void write(byte[] b) throws IOException {
+ contents.write(b);
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) {
+ contents.write(b, off, len);
+ }
+
+ @Override
+ public void write(int b) {
+ contents.write(b);
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (fail) {
Review Comment:
It seems like lot of boilerplate code to simulate this.
Is it possible to use frameworks like `Mockito` (used in other places of the code) to mock the close() of the outputStream or close of the fileIO to throw exception to simulate network error? and don't mock it for the network up case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3079219401
> The issue with the `ParquetFileWriter` mentioned above makes this assertion impossible.
This sounds like something that should be addressed in Parquet first? I don't think I'd be comfortable changing the closing behavior in `ParquetWriter` right now knowing that it could surface a different exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209758736
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,293 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
please fix the license header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209989435
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
+ }
+
+ protected Record toRow(Integer id, String data) {
+ GenericRecord result = GenericRecord.create(table.schema().asStruct());
+ result.set(0, id);
+ result.set(1, data);
+ return result;
+ }
+
+ private static class MyFileWriterFactory extends BaseFileWriterFactory<Record> {
+ protected MyFileWriterFactory(Table table) {
+ super(
+ table,
+ FileFormat.PARQUET,
+ table.schema(),
+ table.sortOrder(),
+ FileFormat.PARQUET,
+ null,
+ table.schema(),
+ table.sortOrder(),
+ table.schema());
+ }
+
+ @Override
+ protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
+ builder.createWriterFunc(GenericParquetWriter::create);
+ }
+
+ @Override
+ protected void configureDataWrite(Avro.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureDataWrite(ORC.DataWriteBuilder builder) {
Review Comment:
Totally.
Unfortunately, I need to implement the abstract methods from the `BaseFileWriterFactory` and these are part of them.
If it works to use the `GenericFileWriterFactory`, as you suggested below, this would also go away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3340852753
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3078557915
> Unfortunately, the [`ParquetFileWriter`](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java), which comes in as a dependency from [parquet-java](https://github.com/apache/parquet-java), seems to have a similar situation that the internal state of the writer is changed before the file has been written successfully. That's why at the moment, simply trying to close the `ParquetWriter` again does not result in the file being written successfully.
Looking at [ParquetFileWriter#close()](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1821-L1833) the `closed` flag is set at the very end. Could you please elaborate on what you mean by "the internal state of the writer is changed before the file has been written successfully"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3078792691
> @nastra Sure! Actually, this time I wasn't writing about the internal `closed`-flag, but now that you point at it, let's discuss it quickly đŸ˜‰: If any code in the try-catch block throws an exception, which would most likely occur in the [`temp.flush()`](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1826), the writer is set to `closed` regardless of the status, as this is done in the `finally`-block.
>
> What I was actually referring to is the field [`state`](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L256), which nicely models the way through the different stages of writing a parquet-file. This one is a bit more tricky than this case though, where just a single line needs to be moved to another location. It tells in which stage the writer currently is and only allowed to advance through the carefully designed stages. But if some IO fails, the internal `state` already advanced into the next stage, making it impossible to go back and try again. If you look at the exception of the attached test, you can see the underlying cause of the final exception:
>
> ```
> Caused by: java.io.IOException: The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: ENDED
> at org.apache.parquet.hadoop.ParquetFileWriter$STATE.error(ParquetFileWriter.java:250)
> at org.apache.parquet.hadoop.ParquetFileWriter$STATE.startBlock(ParquetFileWriter.java:224)
> at org.apache.parquet.hadoop.ParquetFileWriter.startBlock(ParquetFileWriter.java:586)
> at org.apache.iceberg.parquet.ParquetWriter.flushRowGroup(ParquetWriter.java:215)
> ... 100 more
> ```
>
> Being in state `ENDED` suggests that the transition into this state happened [here](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1808), where one can clearly see that the transition should happen only after everything has been processed successfully.
>
> But how should this be solved? As the [end()-method](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1806-L1818) seems to finalize the file with some metadata, it is crucial to not write them for every try a writer tries to close the output. One could try to introduce new entries for the `STATE`-enum which indicate that the writer is about to enter a state (e.g. `STARTING_BLOCK` etc).
The issue with that exception you're seeing where the state being already set to `ENDED` is a result of moving the `closed` flag to the end in `ParquetWriter`, since that flag was also preventing https://github.com/apache/iceberg/blob/main/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java#L261 from being called multiple times.
I haven't fully investigated the issue as I'm currently trying to understand what your expectation in the test is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3078659441
@nastra Sure!
Actually, this time I wasn't writing about the internal `closed`-flag, but now that you point at it, let's discuss it quickly :wink::
If any code in the try-catch block throws an exception, which would most likely occur in the [`temp.flush()`](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1826), the writer is set to `closed` regardless of the status, as this is done in the `finally`-block.
What I was actually referring to is the field [`state`](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L256), which nicely models the way through the different stages of writing a parquet-file.
This one is a bit more tricky than this case though, where just a single line needs to be moved to another location. It tells in which stage the writer currently is and only allowed to advance through the carefully designed stages. But if some IO fails, the internal `state` already advanced into the next stage, making it impossible to go back and try again.
If you look at the exception of the attached test, you can see the underlying cause of the final exception:
```
Caused by: java.io.IOException: The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: ENDED
at org.apache.parquet.hadoop.ParquetFileWriter$STATE.error(ParquetFileWriter.java:250)
at org.apache.parquet.hadoop.ParquetFileWriter$STATE.startBlock(ParquetFileWriter.java:224)
at org.apache.parquet.hadoop.ParquetFileWriter.startBlock(ParquetFileWriter.java:586)
at org.apache.iceberg.parquet.ParquetWriter.flushRowGroup(ParquetWriter.java:215)
... 100 more
```
Being in state `ENDED` suggests that the transition into this state happened [here](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1808), where one can clearly see that the transition should happen only after everything has been processed successfully.
But how should this be solved? As the [end()-method](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1806-L1818) seems to finalize the file with some metadata, it is crucial to not write them for every try a writer tries to close the output. One could try to introduce new entries for the `STATE`-enum which indicate that the writer is about to enter a state (e.g. `STARTING_BLOCK` etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3082920207
Hm, I see your point.
OTOH, this popped up first, being responsible for the silent data loss. Only after we came up with a fix, the second issue popped up. And at least I would always prefer an exception over silent data loss. But in general the issue in parquet-java definitely needs to be addressed too.
I'll postpone working on the suggestions to the test here and file an issue over there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209961540
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
+ }
+
+ protected Record toRow(Integer id, String data) {
+ GenericRecord result = GenericRecord.create(table.schema().asStruct());
+ result.set(0, id);
+ result.set(1, data);
+ return result;
+ }
+
+ private static class MyFileWriterFactory extends BaseFileWriterFactory<Record> {
+ protected MyFileWriterFactory(Table table) {
+ super(
+ table,
+ FileFormat.PARQUET,
+ table.schema(),
+ table.sortOrder(),
+ FileFormat.PARQUET,
+ null,
+ table.schema(),
+ table.sortOrder(),
+ table.schema());
+ }
+
+ @Override
+ protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
+ builder.createWriterFunc(GenericParquetWriter::create);
+ }
+
+ @Override
+ protected void configureDataWrite(Avro.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureDataWrite(ORC.DataWriteBuilder builder) {
Review Comment:
The file name is `TestParquetWriter` using `ORC` here and below places is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209965818
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
Review Comment:
Can you update the testcase to use `GenericFileWriterFactory`, check the code in `TestGenericFileWriterFactory`.
This can avoid lot of redundant code in this class.
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
Review Comment:
Can you update the testcase to use `GenericFileWriterFactory`?, Refer the code in `TestGenericFileWriterFactory`.
This can avoid lot of redundant code in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2210449156
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
Review Comment:
same as mentioned above. Either we're expecting an exception to happen here or not, but we generally want to avoid using try-catch blocks in tests as it's otherwise not clear what exactly is happening and why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209986342
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
Review Comment:
Should be ok. And maybe we can test all formats by making it parameterized test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3314270526
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2209983997
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
Review Comment:
Good hint, thank you!
One issue I see with this it that the `GenericFileWriterFactory` is not `public`, so I would need to move the test to the `org.apache.iceberg.data`-package , instead of having it in `org.apache.iceberg.parquet`. Would that be ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2210158473
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
+ writer.close();
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Error closing writer", e);
+ }
+
+ // The data of the first parquet-file was first written into a byte-buffer and on close should
+ // have been written to file.
+ // But that failed, so the file should not exist:
+ assertThat(PARQUET_FILE_LIST.get(0)).doesNotExist();
+
+ // Simulate that the network is up again:
+ fail = false;
+ // Try again:
+ LOG.info("Trying to close {} again.", writer);
+ try {
+ writer.close();
+ } catch (UncheckedIOException e) {
+ // This error comes from the underlying ParquetFileWriter, which seems to have a similar
+ // problem with setting the internal state too early.
+ if (!"Failed to flush row group".equals(e.getMessage())) {
+ throw e;
+ }
+ // Otherwise, we log and ignore it for now:
+ LOG.warn("The underlying ParquetFileWriter is in an invalid state now:", e);
+ }
+
+ // The writer (or at least one underlying writer) should not be closed:
+ assertThatThrownBy(writer::result)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessage("Cannot get result from unclosed writer");
+ }
+
+ @Override
+ protected FileWriterFactory<Record> newWriterFactory(
+ Schema dataSchema,
+ List<Integer> equalityFieldIds,
+ Schema equalityDeleteRowSchema,
+ Schema positionDeleteRowSchema) {
+ return new MyFileWriterFactory(table);
+ }
+
+ protected Record toRow(Integer id, String data) {
+ GenericRecord result = GenericRecord.create(table.schema().asStruct());
+ result.set(0, id);
+ result.set(1, data);
+ return result;
+ }
+
+ private static class MyFileWriterFactory extends BaseFileWriterFactory<Record> {
+ protected MyFileWriterFactory(Table table) {
+ super(
+ table,
+ FileFormat.PARQUET,
+ table.schema(),
+ table.sortOrder(),
+ FileFormat.PARQUET,
+ null,
+ table.schema(),
+ table.sortOrder(),
+ table.schema());
+ }
+
+ @Override
+ protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
+ builder.createWriterFunc(GenericParquetWriter::create);
+ }
+
+ @Override
+ protected void configureDataWrite(Avro.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureDataWrite(ORC.DataWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+ static final FileIO IO = new FailingLocalFileIo();
+
+ private static class FailingLocalFileIo extends TestTables.LocalFileIO {
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return new FailingLocalOutputFile(Paths.get(path).toAbsolutePath().toFile());
+ }
+ }
+
+ private static class FailingLocalOutputFile implements OutputFile {
+ private final File file;
+
+ private FailingLocalOutputFile(File file) {
+ this.file = file;
+ }
+
+ @Override
+ public PositionOutputStream create() {
+ if (file.exists()) {
+ throw new AlreadyExistsException("File already exists: %s", file);
+ }
+
+ if (!file.getParentFile().isDirectory() && !file.getParentFile().mkdirs()) {
+ throw new RuntimeIOException(
+ "Failed to create the file's directory at %s.", file.getParentFile().getAbsolutePath());
+ }
+
+ PARQUET_FILE_LIST.add(file);
+
+ return new PositionFileOutputStream(file);
+ }
+
+ @Override
+ public PositionOutputStream createOrOverwrite() {
+ if (file.exists() && !file.delete()) {
+ throw new RuntimeIOException("Failed to delete: %s", file);
+ }
+
+ return create();
+ }
+
+ @Override
+ public String location() {
+ return file.toString();
+ }
+
+ @Override
+ public InputFile toInputFile() {
+ return Files.localInput(file);
+ }
+
+ @Override
+ public String toString() {
+ return location();
+ }
+ }
+
+ /**
+ * Keeps all data in memory, before writing it to file when calling close().
+ *
+ * <p>This mimics the behaviour of writing to S3 over network - files are only uploaded when
+ * closing the output.
+ */
+ private static class PositionFileOutputStream extends PositionOutputStream {
+ private final ByteArrayOutputStream contents = new ByteArrayOutputStream();
+ private final File file;
+
+ private PositionFileOutputStream(File file) {
+ this.file = file;
+ }
+
+ @Override
+ public long getPos() {
+ return contents.size();
+ }
+
+ @Override
+ public void write(byte[] b) throws IOException {
+ contents.write(b);
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) {
+ contents.write(b, off, len);
+ }
+
+ @Override
+ public void write(int b) {
+ contents.write(b);
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (fail) {
Review Comment:
Hmm, could be an option.
I quite liked the idea to in the end be able to check if an actual file was created or not and possibly even check its content. This would not be possible with mocking.
My current indicator of success or failure is the result of the wrapping `FanoutDataWriter`, even though the actual issue is in the `ParquetWriter`. Unfortunately, my knowledge of the iceberg-codebase is not that deep (yet). Maybe it is possible to test the actual issue in more isolation, without looking at all the code around it. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#discussion_r2210441772
##########
data/src/test/java/org/apache/iceberg/parquet/TestParquetWriter.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.UncheckedIOException;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.data.BaseFileWriterFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetWriter;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.FanoutDataWriter;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.io.WriterTestBase;
+import org.apache.iceberg.orc.ORC;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestParquetWriter extends WriterTestBase<Record> {
+
+ private static final long TARGET_FILE_SIZE = 128L * 1024 * 1024;
+ private static final Logger LOG = LoggerFactory.getLogger(TestParquetWriter.class);
+
+ private final FileFormat fileFormat = FileFormat.PARQUET;
+
+ private OutputFileFactory fileFactory = null;
+ private static volatile boolean fail = true;
+ private static final List<File> PARQUET_FILE_LIST =
+ Collections.synchronizedList(Lists.newArrayList());
+
+ @BeforeEach
+ void before() {
+ // A lot is already done in TestBase.
+ this.metadataDir = new File(tableDir, "metadata");
+ this.fileFactory =
+ OutputFileFactory.builderFor(table, 1, 1).format(fileFormat).ioSupplier(() -> IO).build();
+ }
+
+ @Test
+ void testParquetWriterWithFailingIO() throws IOException {
+ table.updateSpec().addField(Expressions.ref("data")).commit();
+
+ FileWriterFactory<Record> writerFactory = newWriterFactory(table.schema());
+ FanoutDataWriter<Record> writer =
+ new FanoutDataWriter<>(writerFactory, fileFactory, IO, TARGET_FILE_SIZE);
+
+ PartitionSpec spec = table.spec();
+
+ writer.write(toRow(1, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(3, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(2, "aaa"), spec, partitionKey(spec, "aaa"));
+ writer.write(toRow(4, "bbb"), spec, partitionKey(spec, "bbb"));
+ writer.write(toRow(5, "ccc"), spec, partitionKey(spec, "ccc"));
+
+ try {
Review Comment:
rather than using try-catch blocks in tests what we want is to verify that either an exception has been thrown or not:
```
assertThatThrownBy(writer::close)
.isInstanceOf(UncheckedIOException.class)
.hasMessage("Failed to close current writer");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "fpetersen-gl (via GitHub)" <gi...@apache.org>.
fpetersen-gl commented on PR #13565:
URL: https://github.com/apache/iceberg/pull/13565#issuecomment-3078817940
In the end I'd love to validate that after a second call to `close()` on the writer, all file exists and at best are valid. The issue with the `ParquetFileWriter` mentioned above makes this assertion impossible. That's why I'm currently just testing that the `FanoutDataWriter` (which wraps several `ParquetWriter`s) is not closed yet, as at least one writer is not closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Closing the ParquetWriter in the correct spot [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13565: Closing the ParquetWriter in the correct spot
URL: https://github.com/apache/iceberg/pull/13565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org