You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "zhongyujiang (via GitHub)" <gi...@apache.org> on 2023/11/29 06:51:27 UTC
[PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
zhongyujiang opened a new pull request, #9177:
URL: https://github.com/apache/iceberg/pull/9177
This add a table property `write.delete.path-pos-delete.parquet.row-group-size-bytes` to control the Parquet row-group size of position delete files.
When the position delete file has only `file_path` and `pos` columns, even one million position delete records will only occupy 1~2MB of storage space (assuming they have the same `file_path`):
```text
data-file-row-count: 100000000, pos-delete-row-count : 10000, delete-file-size-bytes: 24617
data-file-row-count: 100000000, pos-delete-row-count : 100000, delete-file-size-bytes: 209195
data-file-row-count: 100000000, pos-delete-row-count : 500000, delete-file-size-bytes: 922813
data-file-row-count: 100000000, pos-delete-row-count : 1000000, delete-file-size-bytes: 1625049
```
However, currently the default value of Parquet row-group size of position delete is 128MB, which is too big. The reader must read all pos delete to find the relevant position deletes because all delete records are written to one row-group.
So this adds a property to control the row-group size of position delete file, it will be used when position deletes has only `file_path` and `pos` colunmns. The default is set to 2MB to separate position deletes that should be applied to different data files into different row-groups as much as possible, so that we can take advantage of the row-group filter to filter out unreleated position delete records when reading.
closes #9149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#issuecomment-2412570798
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#issuecomment-1831325885
@aokolnychyi @szehon-ho could you please take a look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#issuecomment-1845189563
@rdblue @RussellSpitzer would you mind take a look at this? Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#discussion_r1408832480
##########
spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.infra.Blackhole;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A benchmark that evaluates the non-vectorized read and vectorized read with pos-delete in the
+ * Spark data source for Iceberg.
+ *
+ * <p>This class uses a dataset with a flat schema. To run this benchmark for spark-3.5: <code>
+ * ./gradlew -DsparkVersions=3.5 :iceberg-spark:iceberg-spark-3.5:jmh
+ * -PjmhIncludeRegex=IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized
+ * -PjmhOutputPath=benchmark/iceberg-source-parquet-pos-delete-row-group-filter-benchmark-result.txt
+ * </code>
+ */
+public class IcebergSourceParquetPosDeleteRowGroupFilterBenchmark
+ extends IcebergSourceDeleteBenchmark {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.class);
+
+ private static final Schema DELETE_SCHEMA =
+ new Schema(MetadataColumns.DELETE_FILE_PATH, MetadataColumns.DELETE_FILE_POS);
+
+ @Param({"2", "4", "128" /* default size */})
+ private int rowGroupSizeMB;
+
+ @Param({"3", "5"})
+ private int numDataFiles;
+
+ private DeleteFile posDeleteFile;
+
+ @Override
+ protected void appendData() throws IOException {
+ for (int fileNum = 1; fileNum <= numDataFiles; fileNum++) {
+ writeData(fileNum);
+ }
+
+ table()
+ .updateProperties()
+ .set(
+ TableProperties.PATH_POS_DELETE_PARQUET_ROW_GROUP_SIZE_BYTES,
+ String.valueOf(rowGroupSizeMB * 1024 * 1024))
+ .commit();
+
+ // add pos-deletes
+ table().refresh();
+ List<CharSequence> filePaths =
+ Streams.stream(table().newScan().planFiles())
+ .map(task -> task.file().path())
+ .collect(Collectors.toList());
+
+ writePosDeletes(filePaths, NUM_ROWS, 0.10);
+
+ table().refresh();
+ posDeleteFile =
+ Iterables.getOnlyElement(table().currentSnapshot().addedDeleteFiles(table().io()));
+ List<Long> offsets = posDeleteFile.splitOffsets();
+
+ LOG.info(
+ "row-group-size-MB: {}, position delete file size in bytes: {}, number row-groups: {}",
+ rowGroupSizeMB,
+ posDeleteFile.fileSizeInBytes(),
+ offsets.size());
Review Comment:
This is the row-group split status of position delete files when using different row-group sizes, the actual row-group size might be smaller than the set value because Parquet will make judgments based on the unflushed data size when splitting row-groups:
```
row-group-size-MB: 128, position delete file size in bytes: 9921119, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 9921030, number row-groups: 3
row-group-size-MB: 2, position delete file size in bytes: 9925059, number row-groups: 8
row-group-size-MB: 128, position delete file size in bytes: 5951966, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 5952190, number row-groups: 2
row-group-size-MB: 2, position delete file size in bytes: 5955593, number row-groups: 5
```
And below is the benchmark result:
```
Benchmark (numDataFiles) (rowGroupSizeMB) Mode Cnt Score Error Units
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 2 ss 5 4.853 ± 0.632 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 4 ss 5 5.794 ± 0.379 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 128 ss 5 6.207 ± 0.278 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 2 ss 5 8.758 ± 0.610 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 4 ss 5 10.270 ± 0.586 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 128 ss 5 13.734 ± 0.433 s/op
```
##########
spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.infra.Blackhole;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A benchmark that evaluates the non-vectorized read and vectorized read with pos-delete in the
+ * Spark data source for Iceberg.
+ *
+ * <p>This class uses a dataset with a flat schema. To run this benchmark for spark-3.5: <code>
+ * ./gradlew -DsparkVersions=3.5 :iceberg-spark:iceberg-spark-3.5:jmh
+ * -PjmhIncludeRegex=IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized
+ * -PjmhOutputPath=benchmark/iceberg-source-parquet-pos-delete-row-group-filter-benchmark-result.txt
+ * </code>
+ */
+public class IcebergSourceParquetPosDeleteRowGroupFilterBenchmark
+ extends IcebergSourceDeleteBenchmark {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.class);
+
+ private static final Schema DELETE_SCHEMA =
+ new Schema(MetadataColumns.DELETE_FILE_PATH, MetadataColumns.DELETE_FILE_POS);
+
+ @Param({"2", "4", "128" /* default size */})
+ private int rowGroupSizeMB;
+
+ @Param({"3", "5"})
+ private int numDataFiles;
+
+ private DeleteFile posDeleteFile;
+
+ @Override
+ protected void appendData() throws IOException {
+ for (int fileNum = 1; fileNum <= numDataFiles; fileNum++) {
+ writeData(fileNum);
+ }
+
+ table()
+ .updateProperties()
+ .set(
+ TableProperties.PATH_POS_DELETE_PARQUET_ROW_GROUP_SIZE_BYTES,
+ String.valueOf(rowGroupSizeMB * 1024 * 1024))
+ .commit();
+
+ // add pos-deletes
+ table().refresh();
+ List<CharSequence> filePaths =
+ Streams.stream(table().newScan().planFiles())
+ .map(task -> task.file().path())
+ .collect(Collectors.toList());
+
+ writePosDeletes(filePaths, NUM_ROWS, 0.10);
+
+ table().refresh();
+ posDeleteFile =
+ Iterables.getOnlyElement(table().currentSnapshot().addedDeleteFiles(table().io()));
+ List<Long> offsets = posDeleteFile.splitOffsets();
+
+ LOG.info(
+ "row-group-size-MB: {}, position delete file size in bytes: {}, number row-groups: {}",
+ rowGroupSizeMB,
+ posDeleteFile.fileSizeInBytes(),
+ offsets.size());
Review Comment:
This is the row-group split status of position delete files when using different row-group sizes, the actual row-group size might be smaller than the set value because Parquet will make judgments based on the unflushed data size when splitting row-groups:
```
row-group-size-MB: 128, position delete file size in bytes: 9921119, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 9921030, number row-groups: 3
row-group-size-MB: 2, position delete file size in bytes: 9925059, number row-groups: 8
row-group-size-MB: 128, position delete file size in bytes: 5951966, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 5952190, number row-groups: 2
row-group-size-MB: 2, position delete file size in bytes: 5955593, number row-groups: 5
```
And below is the benchmark result:
```
Benchmark (numDataFiles) (rowGroupSizeMB) Mode Cnt Score Error Units
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 2 ss 5 4.853 ± 0.632 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 4 ss 5 5.794 ± 0.379 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 128 ss 5 6.207 ± 0.278 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 2 ss 5 8.758 ± 0.610 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 4 ss 5 10.270 ± 0.586 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 128 ss 5 13.734 ± 0.433 s/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#discussion_r1408832480
##########
spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.infra.Blackhole;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A benchmark that evaluates the non-vectorized read and vectorized read with pos-delete in the
+ * Spark data source for Iceberg.
+ *
+ * <p>This class uses a dataset with a flat schema. To run this benchmark for spark-3.5: <code>
+ * ./gradlew -DsparkVersions=3.5 :iceberg-spark:iceberg-spark-3.5:jmh
+ * -PjmhIncludeRegex=IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized
+ * -PjmhOutputPath=benchmark/iceberg-source-parquet-pos-delete-row-group-filter-benchmark-result.txt
+ * </code>
+ */
+public class IcebergSourceParquetPosDeleteRowGroupFilterBenchmark
+ extends IcebergSourceDeleteBenchmark {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.class);
+
+ private static final Schema DELETE_SCHEMA =
+ new Schema(MetadataColumns.DELETE_FILE_PATH, MetadataColumns.DELETE_FILE_POS);
+
+ @Param({"2", "4", "128" /* default size */})
+ private int rowGroupSizeMB;
+
+ @Param({"3", "5"})
+ private int numDataFiles;
+
+ private DeleteFile posDeleteFile;
+
+ @Override
+ protected void appendData() throws IOException {
+ for (int fileNum = 1; fileNum <= numDataFiles; fileNum++) {
+ writeData(fileNum);
+ }
+
+ table()
+ .updateProperties()
+ .set(
+ TableProperties.PATH_POS_DELETE_PARQUET_ROW_GROUP_SIZE_BYTES,
+ String.valueOf(rowGroupSizeMB * 1024 * 1024))
+ .commit();
+
+ // add pos-deletes
+ table().refresh();
+ List<CharSequence> filePaths =
+ Streams.stream(table().newScan().planFiles())
+ .map(task -> task.file().path())
+ .collect(Collectors.toList());
+
+ writePosDeletes(filePaths, NUM_ROWS, 0.10);
+
+ table().refresh();
+ posDeleteFile =
+ Iterables.getOnlyElement(table().currentSnapshot().addedDeleteFiles(table().io()));
+ List<Long> offsets = posDeleteFile.splitOffsets();
+
+ LOG.info(
+ "row-group-size-MB: {}, position delete file size in bytes: {}, number row-groups: {}",
+ rowGroupSizeMB,
+ posDeleteFile.fileSizeInBytes(),
+ offsets.size());
Review Comment:
This is the row-group split status of position delete files when using different row-group sizes, the actual row-group size might be smaller than the set value because Parquet makes judgments based on the unflushed data size when splitting row-groups:
```
row-group-size-MB: 128, position delete file size in bytes: 9921119, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 9921030, number row-groups: 3
row-group-size-MB: 2, position delete file size in bytes: 9925059, number row-groups: 8
row-group-size-MB: 128, position delete file size in bytes: 5951966, number row-groups: 1
row-group-size-MB: 4, position delete file size in bytes: 5952190, number row-groups: 2
row-group-size-MB: 2, position delete file size in bytes: 5955593, number row-groups: 5
```
And below is the benchmark result:
```
Benchmark (numDataFiles) (rowGroupSizeMB) Mode Cnt Score Error Units
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 2 ss 5 4.853 ± 0.632 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 4 ss 5 5.794 ± 0.379 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 3 128 ss 5 6.207 ± 0.278 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 2 ss 5 8.758 ± 0.610 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 4 ss 5 10.270 ± 0.586 s/op
IcebergSourceParquetPosDeleteRowGroupFilterBenchmark.readIcebergVectorized 5 128 ss 5 13.734 ± 0.433 s/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9177: Parquet: Add a table property to control the Parquet row-group size of position delete files
URL: https://github.com/apache/iceberg/pull/9177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Add a table property to control the Parquet row-group size of position delete files [iceberg]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #9177:
URL: https://github.com/apache/iceberg/pull/9177#issuecomment-1831405339
The [CI failure](https://github.com/apache/iceberg/actions/runs/7029032258/job/19125970118#step:6:214) seems unreleated:
```
> Task :iceberg-flink:iceberg-flink-1.17:test
TestIcebergSourceWithWatermarkExtractor > testThrottling FAILED
java.lang.AssertionError:
Expecting
<CompletableFuture[Failed with the following stack trace:
java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
at org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:400)
at org.apache.flink.streaming.api.operators.collect.CollectCoordinationResponse.getResults(CollectCoordinationResponse.java:84)
at org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.addResults(AbstractCollectResultBuffer.java:127)
at org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.dealWithResponse(AbstractCollectResultBuffer.java:91)
at org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:147)
at org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
at org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
at org.apache.iceberg.flink.source.TestIcebergSourceWithWatermarkExtractor.lambda$waitForRecords$6(TestIcebergSourceWithWatermarkExtractor.java:405)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
]>
to be completed within 2M.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org