You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2023/10/11 12:33:29 UTC
[PR] Core: Enable column statistics filtering after planning [iceberg]
pvary opened a new pull request, #8803:
URL: https://github.com/apache/iceberg/pull/8803
Based on our discussion on the dev list, I have created the PR which makes possible to narrow down the retained column statistics in the `ScanTask` returned from planning.
For reference the discussion: https://lists.apache.org/thread/pcfpztld5gfpdvm1dy4l84xfl6odxhw8
The PR makes it possible to set the `includeColumnStats` for a `Scan`. The resulting `ScanTask`s will contain column statistics for the specific columnIds only, omitting statistics which might be present in the metadata files, but not specifically requested by the user.
The PR consists of 3 main parts:
1. Interface changes:
- `Scan.includeColumnStats` to set the required columnIds
- `ContentFile.copyWithSpecificStats` to provide an interface for the stat removal when copying the file objects
2. Core changes:
- Implementation of the `BaseFile` constructor which takes care of the statistics filtering, and making sure that the other implementations are using this method as well.
- Propagating the `columnStatsToInclude` filed through the different scan implementations, and putting it into the `TableScanContext`.
- Adding a new property to the `ManifestGroup` builder to store the `columnStatsToKeep`. This class is responsible for the final copy of the `DataFiles` where we remove the statistics which are not needed.
- Added tests to check that the statistics removal is working as expected.
3. Flink changes:
- Adding a new `FlinkReadOption` to set which column stats we should keep: `column-stats-to-keep`
- Minimal Flink `ScanContext` and Planner changes to propagate the values
- Updated the documentation for the Flink Source
- Added tests to check that the statistics removal is working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1355087396
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/enumerator/TestContinuousSplitPlannerImpl.java:
##########
@@ -533,6 +533,116 @@ public void testMaxPlanningSnapshotCount() throws Exception {
thirdResult, snapshot1, snapshot2, ImmutableSet.of(dataFile2.path().toString()));
}
+ @Test
+ public void testTableScanNoStats() throws Exception {
+ appendTwoSnapshots();
+
+ ScanContext scanContext =
+ ScanContext.builder()
+ .includeColumnStats(false)
+ .startingStrategy(StreamingStartingStrategy.TABLE_SCAN_THEN_INCREMENTAL)
+ .build();
+ ContinuousSplitPlannerImpl splitPlanner =
+ new ContinuousSplitPlannerImpl(tableResource.tableLoader().clone(), scanContext, null);
+
+ ContinuousEnumerationResult initialResult = splitPlanner.planSplits(null);
+ Assert.assertEquals(1, initialResult.splits().size());
Review Comment:
This is a few new test in a test class where different assertion methods are used. I think we should migrate them in one PR, and until that stick to the style which is used in the actual test.
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -191,6 +192,11 @@ public DataFile copy() {
throw new UnsupportedOperationException("Not implemented: copy");
}
+ @Override
+ public DataFile copyWithSpecificStats(Collection<Integer> statsToKeep) {
Review Comment:
Yeah.. removed.
Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682090
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT columnsToKeepStats(Set<Integer> columnsToKeepStats) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1780243850
I took a quick look. My original guess was to do this in `ManifestReader` but it seems Peter is right: it should be sufficient to do this in `ManifestGroup` as we are using either `entries()` or `liveEntries()` and call `copy()` in `ManifestGroup`, rather than relying on `iterator()`.
I'll check with fresh eyes tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1372831403
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
I did some testing with Parquet files:
- Used `TestParquet` to check what `Metrics` do we generate when we have only `null` values:
```
@Test
public void testNullMetricsForEmptyColumns() throws IOException {
Schema schema = new Schema(optional(1, "longCol", Types.LongType.get()),
optional(2, "nullCol", Types.LongType.get()));
File file = createTempFile(temp);
List<GenericData.Record> records = Lists.newArrayListWithCapacity(1);
org.apache.avro.Schema avroSchema = AvroSchemaUtil.convert(schema.asStruct());
GenericData.Record smallRecord = new GenericData.Record(avroSchema);
smallRecord.put("longCol", 1L);
records.add(smallRecord);
write(
file,
schema,
Collections.emptyMap(),
ParquetAvroWriter::buildWriter,
records.toArray(new GenericData.Record[] {}));
InputFile inputFile = Files.localInput(file);
Metrics metrics = ParquetUtil.fileMetrics(inputFile, MetricsConfig.getDefault());
assertThat(metrics.nullValueCounts()).hasSize(2);
assertThat(metrics.lowerBounds()).hasSize(1).containsKey(1);
assertThat(metrics.upperBounds()).hasSize(1).containsKey(1);
}
```
Notice, that the size of the `lowerBounds` and the `upperBounds` map is 1. There is no `null` value for column '2'
- Just to double check, I have tested how we read files with missing column metrics (like the ones created above). I have used `TestManifestReaderStats` for this:
```
@Test
public void testReadFileWithMissingStatsIncludesFullStats() throws IOException {
Metrics missingStats =
new Metrics(
3L, null,
ImmutableMap.of(3, 3L, 4, 4L),
ImmutableMap.of(3, 3L, 4, 0L),
ImmutableMap.of(3, 0L, 4, 1L),
ImmutableMap.of(4, Conversions.toByteBuffer(Types.StringType.get(), "a")),
ImmutableMap.of(4, Conversions.toByteBuffer(Types.StringType.get(), "a")));
DataFile fileWithMissingStats =
DataFiles.builder(SPEC)
.withPath(FILE_PATH)
.withFileSizeInBytes(10)
.withPartitionPath("data_bucket=0") // easy way to set partition data for now
.withRecordCount(3)
.withMetrics(missingStats)
.build();
ManifestFile manifest = writeManifest(1000L, fileWithMissingStats);
try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, FILE_IO)) {
CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
ManifestEntry<DataFile> entry = entries.iterator().next();
DataFile dataFile = entry.file();
Assert.assertEquals(3, dataFile.recordCount());
assertThat(dataFile.lowerBounds()).hasSize(1).containsKey(4);
assertThat(dataFile.upperBounds()).hasSize(1).containsKey(4);
}
}
```
Notice, that the size of the `lowerBounds` and the `upperBounds` map is 1. There is no `null` value for column '3'
So I think the actual phrasing of the documentation is just not clean enough, as the map does not contain `null` values, it just does not contain the keys. While putting `null` value to a map is certainly possible, it should be used only in very specific cases, as it is very error-prone (see our discussion above 😄). If our goal would be to identify the case where all of the values in the given column are `null`, we can use the other stats to archive the result.
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
I did some testing with Parquet files:
- Used `TestParquet` to check what `Metrics` do we generate when we have only `null` values:
```
@Test
public void testNullMetricsForEmptyColumns() throws IOException {
Schema schema = new Schema(optional(1, "longCol", Types.LongType.get()),
optional(2, "nullCol", Types.LongType.get()));
File file = createTempFile(temp);
List<GenericData.Record> records = Lists.newArrayListWithCapacity(1);
org.apache.avro.Schema avroSchema = AvroSchemaUtil.convert(schema.asStruct());
GenericData.Record smallRecord = new GenericData.Record(avroSchema);
smallRecord.put("longCol", 1L);
records.add(smallRecord);
write(
file,
schema,
Collections.emptyMap(),
ParquetAvroWriter::buildWriter,
records.toArray(new GenericData.Record[] {}));
InputFile inputFile = Files.localInput(file);
Metrics metrics = ParquetUtil.fileMetrics(inputFile, MetricsConfig.getDefault());
assertThat(metrics.nullValueCounts()).hasSize(2);
assertThat(metrics.lowerBounds()).hasSize(1).containsKey(1);
assertThat(metrics.upperBounds()).hasSize(1).containsKey(1);
}
```
Notice, that the size of the `lowerBounds` and the `upperBounds` map is 1. There is no `null` value for column '2'
- Just to double check, I have tested how we read files with missing column metrics (like the ones created above). I have used `TestManifestReaderStats` for this:
```
@Test
public void testReadFileWithMissingStatsIncludesFullStats() throws IOException {
Metrics missingStats =
new Metrics(
3L, null,
ImmutableMap.of(3, 3L, 4, 4L),
ImmutableMap.of(3, 3L, 4, 0L),
ImmutableMap.of(3, 0L, 4, 1L),
ImmutableMap.of(4, Conversions.toByteBuffer(Types.StringType.get(), "a")),
ImmutableMap.of(4, Conversions.toByteBuffer(Types.StringType.get(), "a")));
DataFile fileWithMissingStats =
DataFiles.builder(SPEC)
.withPath(FILE_PATH)
.withFileSizeInBytes(10)
.withPartitionPath("data_bucket=0") // easy way to set partition data for now
.withRecordCount(3)
.withMetrics(missingStats)
.build();
ManifestFile manifest = writeManifest(1000L, fileWithMissingStats);
try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, FILE_IO)) {
CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
ManifestEntry<DataFile> entry = entries.iterator().next();
DataFile dataFile = entry.file();
Assert.assertEquals(3, dataFile.recordCount());
assertThat(dataFile.lowerBounds()).hasSize(1).containsKey(4);
assertThat(dataFile.upperBounds()).hasSize(1).containsKey(4);
}
}
```
Notice, that the size of the `lowerBounds` and the `upperBounds` map is 1. There is no `null` value for column '3'
So I think the actual phrasing of the documentation is just not clean enough, as the map does not contain `null` values, it just does not contain the keys. While putting `null` value to a map is certainly possible, it should be used only in very specific cases, as it is very error-prone (see our discussion above 😄). If our goal would be to identify the case where all of the values in the given column are `null`, we can use the other stats to archive the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371944613
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
Review Comment:
The extra null check is needed. If we have a non-empty `columnIds`, but an empty `map`, then I do not want to do the filtering, and stuff. We just return `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375314743
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> columnsToKeepStats) {
Review Comment:
I think the object name is confusing. It doesn't need to match the method name and should be more straightforward. How about `requestedColumns` or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682326
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> columnsToKeepStats) {
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> columnsToKeepStats) {
+ super(toCopy, copyStats, columnsToKeepStats);
}
/** Constructor for Java serialization. */
GenericDataFile() {}
@Override
public DataFile copyWithoutStats() {
- return new GenericDataFile(this, false /* drop stats */);
+ return new GenericDataFile(this, false /* drop stats */, ImmutableSet.of());
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> columnsToKeepStats) {
+ return new GenericDataFile(this, true, columnsToKeepStats);
}
@Override
public DataFile copy() {
- return new GenericDataFile(this, true /* full copy */);
+ return new GenericDataFile(this, true /* full copy */, ImmutableSet.of());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1389360789
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +190,25 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
Review Comment:
Had to move the proposed `SerializableMap` changes back to the `BaseFile` to avoid changing the default serialization of the `SerializableMap`:
```
* Just this break:
./gradlew :iceberg-core:revapiAcceptBreak --justification "{why this break is ok}" \
--code "java.class.defaultSerializationChanged" \
--old "class org.apache.iceberg.util.SerializableMap<K extends java.lang.Object, V extends java.lang.Object>" \
--new "class org.apache.iceberg.util.SerializableMap<K extends java.lang.Object, V extends java.lang.Object>"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1391927184
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,11 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
+ org.apache.iceberg:iceberg-core:
+ - code: "java.field.serialVersionUIDChanged"
Review Comment:
While I think it should be fine, here is an idea. Java comes with `serialver` utility that allows us to generate the version UID prior to the change in this PR. We can use that value instead of `1L` to be fully compatible. We don't modify the serialization of this class, we just missed to assign `serialVersionUID`. If we can recover the default value, we shouldn't worry about compatibility.
Here is the value I got locally:
```
cd core/build/classes/java/main
serialver org.apache.iceberg.util.SerializableMap
org.apache.iceberg.util.SerializableMap: private static final long serialVersionUID = -3377238354349859240L;
```
Could you double check, @pvary? If not, we can keep it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392372415
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -78,7 +78,8 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.select(scanColumns())
.filterData(filter())
.filterManifestEntries(entry -> changelogSnapshotIds.contains(entry.snapshotId()))
- .ignoreExisting();
+ .ignoreExisting()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/DataScan.java:
##########
@@ -55,7 +55,8 @@ protected ManifestGroup newManifestGroup(
.filterData(filter())
.specsById(table().specs())
.scanMetrics(scanMetrics())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1389233962
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +190,25 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
Review Comment:
I went back to my original solution (the most efficient one), and used the `SerializableMap` changes you have suggested.
Also changed a bit of the `SerializableMap` constructor to address @stevenzwu previous comments.
In case the source map contains `null` values, then we do not copy them to the target map. We should use the following code to correctly handle `null` values:
```
for (K key : keys) {
if (map.containsKey(key)) {
filteredMap.put(key, map.get(key));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1384763093
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
@rdblue, @aokolnychyi, @stevenzwu: I would appreciate your thoughts here.
- Do you have any better idea how to handle `ContentFile.copyWithStats` other than keeping them as Ids?
- Is it ok, to keep it as a `Set`, or shall I turn this to a `Collection` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392371790
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +188,30 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
+ if (requestedColumnIds == null) {
Review Comment:
Applied the solution suggested by you
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalAppendScan.java:
##########
@@ -83,7 +83,8 @@ private CloseableIterable<FileScanTask> appendFilesFromSnapshots(List<Snapshot>
snapshotIds.contains(manifestEntry.snapshotId())
&& manifestEntry.status() == ManifestEntry.Status.ADDED)
.specsById(table().specs())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1809974009
@rdblue: I have fixed the changes requested by you. If you have any further comments, please leave a review.
@aokolnychyi did another throughout review and applied most of his suggested changes.
So with 2 +1's, I would like to merge this change in the next few days.
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369761206
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,30 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param fullCopy whether to copy all fields or to drop column-level stats.
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(GenericDataFile toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/GenericDeleteFile.java:
##########
@@ -67,23 +69,30 @@ class GenericDeleteFile extends BaseFile<DeleteFile> implements DeleteFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param fullCopy whether to copy all fields or to drop column-level stats.
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDeleteFile(GenericDeleteFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDeleteFile(GenericDeleteFile toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -381,19 +390,22 @@ static class TaskContext {
private final DeleteFileIndex deletes;
private final ResidualEvaluator residuals;
private final boolean dropStats;
+ private final Set<Integer> statsToKeep;
private final ScanMetrics scanMetrics;
TaskContext(
PartitionSpec spec,
DeleteFileIndex deletes,
ResidualEvaluator residuals,
boolean dropStats,
+ Set<Integer> statsToKeep,
Review Comment:
Fixed everywhere (I hope, will check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369762281
##########
docs/flink-configuration.md:
##########
@@ -130,6 +130,7 @@ env.getConfig()
| streaming | connector.iceberg.streaming | N/A | false | Sets whether the current task runs in streaming or batch mode.
|
| monitor-interval | connector.iceberg.monitor-interval | N/A | 60s | Monitor interval to discover splits from new snapshots. Applicable only for streaming read.
|
| include-column-stats | connector.iceberg.include-column-stats | N/A | false | Create a new scan from this that loads the column stats with each data file. Column stats include: value count, null value count, lower bounds, and upper bounds.
|
+| column-stats-to-keep | connector.iceberg.column-stats-to-keep | N/A | empty | Create a new scan from this that loads the column stats with each data file for the specified column ids. Column stats include: value count, null value count, lower bounds, and upper bounds.
|
Review Comment:
Removed as per another comment
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
Review Comment:
Fixed everywhere (I hope, will check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1355072299
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,15 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
Review Comment:
Good point!
Removed!
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375315072
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> columnsToKeepStats) {
+ super(toCopy, copyStats, columnsToKeepStats);
}
/** Constructor for Java serialization. */
GenericDataFile() {}
@Override
public DataFile copyWithoutStats() {
- return new GenericDataFile(this, false /* drop stats */);
+ return new GenericDataFile(this, false /* drop stats */, ImmutableSet.of());
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> columnsToKeepStats) {
+ return new GenericDataFile(this, true, columnsToKeepStats);
}
@Override
public DataFile copy() {
- return new GenericDataFile(this, true /* full copy */);
+ return new GenericDataFile(this, true /* full copy */, ImmutableSet.of());
Review Comment:
Why does this pass an empty set for a full copy? Those two things conflict with one another. If you intend to allow not passing a set of column IDs (which is a good idea) then pass a default that makes sense to turn off the filter, like `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375314451
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT columnsToKeepStats(Set<Integer> columnsToKeepStats) {
Review Comment:
I think this should be `includeColumnStatsFor` or `includeStatsForColumns` to align with the readability conventions and match the `includeColumnStats` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375314451
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT columnsToKeepStats(Set<Integer> columnsToKeepStats) {
Review Comment:
I think this should be `keepStatsFor` to align with the readability conventions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1388806220
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> requestedColumnIds) {
Review Comment:
I spent a bit more time on this one. I do think we either drop support for `null` as a valid value here or drop the utility to copy. It is up to you which approach to pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365196098
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,19 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param statsToKeep the collection of the column ids for the columns which stats are kept
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithSpecificStats(Collection<Integer> statsToKeep) {
Review Comment:
Done
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +77,20 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columns column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT includeColumnStats(Collection<Integer> columns) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392376668
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -154,6 +156,12 @@ ManifestGroup caseSensitive(boolean newCaseSensitive) {
return this;
}
+ ManifestGroup columnsToKeepStats(Set<Integer> newColumnsToKeepStats) {
+ this.columnsToKeepStats =
+ newColumnsToKeepStats == null ? null : Sets.newHashSet(newColumnsToKeepStats);
Review Comment:
Kept as it is more consistent with the other implementations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392372882
##########
core/src/main/java/org/apache/iceberg/DataTableScan.java:
##########
@@ -76,7 +76,8 @@ public CloseableIterable<FileScanTask> doPlanFiles() {
.filterData(filter())
.specsById(table().specs())
.scanMetrics(scanMetrics())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382661567
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
@aokolnychyi the proposed interfaces for `Scan` and `ContentFile` make sense to me.
> A collection with a single element * means we will call copy() on files.
> Null or empty collection means we will call copyWithoutStats() on files.
Regarding the collection of a single `*` element, it feels a little tacky to me. The util method/class takes the two configs from the `TableScanContext` and implements the copy logic.
> All of the logic above can be incapsulated in a single copy method in our base scan.
This `ContentFileUtil` was intended for that purpose of incapsulating all those logic. Note that `ManifestGroup` class also leverages this util method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1373440573
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
thanks for the unit test code. I am still not comfortable of making the assumption as the Iceberg spec clearly said null stats value is possible. what if Trino parquet writer produces min/max stats with null value.
```
Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]
```
Unless we are going to clarify/update the spec, I feel it is safer to do `map.containsKey(key)`.
Maybe @RussellSpitzer and @rdblue can chime in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371955308
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +177,10 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Set<Integer> columnsToKeepStats) {
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375315318
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
This API does not use column IDs, it uses column names. This method should not accept a collection of IDs and should accept strings and convert instead.
Also, the convention is to use `Collection<String>` rather than `Set` and to add a `String...` method for convenience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1362401859
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -177,4 +191,26 @@ default Long fileSequenceNumber() {
default F copy(boolean withStats) {
return withStats ? copy() : copyWithoutStats();
}
+
+ /**
+ * Copies this file (potentially with or without specific column stats). Manifest readers can
+ * reuse file instances; use this method to copy data when collecting files from tasks.
+ *
+ * @param withStats Will copy this file without file stats if set to <code>false</code>.
+ * @param statsToKeep Will keep stats only for these columns. Not used if <code>withStats</code>
+ * is set to <code>false</code>.
+ * @return a copy of this data file. If "withStats" is set to <code>false</code> the file will not
+ * contain lower bounds, upper bounds, value counts, null value counts, or nan value counts.
+ * If "withStats" is set to <code>true</code> and the "statsToKeep" is not empty then only
+ * specific column stats will be kept.
+ */
+ default F copy(boolean withStats, Collection<Integer> statsToKeep) {
Review Comment:
is this needed in this interface? `copyWithSpecificStats` from line 177 should imply `withStats` to be true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382325507
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
+1.
We have `select` that accepts `Collection<String> columns` below. I understand using IDs is straightforward but someone has to do the conversion from name to ID and we better do it ourselves behind this API. If an external long-running process needs to ensure the ID did not change, it can that check outside.
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
+1.
We have `select` that accepts `Collection<String> columns` below. I understand using IDs is straightforward but someone has to do the conversion from name to ID and we better do it ourselves behind this API. If an external long-running process needs to ensure the ID did not change, it can perform that check outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382326565
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,21 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param columnsToKeepStats the set of the column ids for the columns which stats are kept. If
+ * the columnsToKeepStats set is empty or <code>null</code>, then all column stats will be
+ * kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> columnsToKeepStats) {
Review Comment:
If we decide to do [this](https://github.com/apache/iceberg/pull/8803#discussion_r1375314743) rename, we should probably be consistent here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365197840
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -186,12 +189,23 @@ public PartitionData copy() {
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (statsToKeep == null || statsToKeep.isEmpty()) {
Review Comment:
I did it this way, to save 5 checks, but the code is much nicer if we hide the checks in the method, so refactored
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -59,6 +60,11 @@ public boolean returnColumnStats() {
return false;
}
+ @Value.Default
+ public Collection<Integer> returnColumnStatsToInclude() {
Review Comment:
done
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -125,6 +131,16 @@ TableScanContext shouldReturnColumnStats(boolean returnColumnStats) {
.build();
}
+ TableScanContext shouldReturnSpecificColumnStats(Collection<Integer> columStatsToInclude) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1367939105
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
+ return split(
Review Comment:
Regarding schema evolution (like field rename), users have to adjust the name reference. this is also applicable to filter expressions or column projections. Users typically reference fields by name. I still treat field id as internal identifier to ensure safe schema evolution. it is not friendly to ask users to provide filed ids for filter, projection, etc.
In another thought, I am also not sure we need to expose this to users directly. The only usage of column stats is the watermark alignment. we will need to expose `enableWatermarkAlignment(String watermarkField)` anyway. there is probably no need to expose this directly, at least not yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369754595
##########
api/src/test/java/org/apache/iceberg/TestHelpers.java:
##########
@@ -662,6 +663,11 @@ public DataFile copyWithoutStats() {
return this;
}
+ @Override
+ public DataFile copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +176,10 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369761988
##########
core/src/main/java/org/apache/iceberg/V1Metadata.java:
##########
@@ -485,6 +486,11 @@ public DataFile copy() {
return wrapped.copy();
}
+ @Override
+ public DataFile copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/V2Metadata.java:
##########
@@ -560,6 +561,11 @@ public F copy() {
throw new UnsupportedOperationException("Cannot copy IndexedDataFile wrapper");
}
+ @Override
+ public F copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1385865215
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +77,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param requestedColumns column names for which to keep the stats. If <code>null</code> then all
Review Comment:
I am not sure how I feel about supporting `null` here. Let me think.
Also, won't the current implementation throw an exception if we pass `null`?
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +190,25 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
Review Comment:
I wonder about the performance and generated extra objects given that it is such a critical place that's invoked in a tight loop. We can have millions of files. Will need to check this with fresh eyes tomorrow.
We should run our JMH benchmarks before merging. If the difference is visible, we can offer `copyOf` methods that would copy only selected keys in our custom maps.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> requestedColumnIds) {
Review Comment:
Also, if we want to support `null` as a valid value, do we still need `ContentFileUtil`?
```
copyWithStats(null) -> copy with all stats
copyWithStats(empty) -> copy without stats
copyWithStats(someIds) -> copy with some stats
```
It seems we need to either support `null` here or have `ContenFileUtil`, is there a reason to have both? I am not sure what I prefer more, though.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> requestedColumnIds) {
Review Comment:
I think it is fine to use IDs at this level as `ContenFile` references IDs everywhere. I agree with converting column names to IDs in scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1770262566
> I understand the current code is for metadata column selection/projection, not the columns selected to include stats
My understanding is that the planning might need stats which are not required by the user.
Based on this for changing the stat retrieval here:
- We need some more complicated decision tree for which column stats to keep (union of the user requested stats, and the planning required stats)
- We (might) need to do the selective copy of the stats later again to remove the planning required stats which are not requested by the user. This might have a detrimental performance impact.
This change improves on memory consumption anyway. If we think that we need even more improvement and we accept the extra complexity we can add this feature later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1768023880
> @pvary I think we probably want to push the `copyStatsForColumns` down to ManifestReader. https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestReader.java#L299
That is for reading the data from the manifest file.
If we want at statistics for at least one column, then the manifest file reading schema should contain the stat fields, like:
```
private static final Set<String> STATS_COLUMNS =
ImmutableSet.of(
"value_counts",
"null_value_counts",
"nan_value_counts",
"lower_bounds",
"upper_bounds",
"record_count");
```
So we can not do filtering here. We need to read the stat fields from the manifest file, and then filter later for columns where we do not need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1766844420
@pvary I think we probably want to push the `copyStatsForColumns` down to ManifestReader. https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestReader.java#L299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365196782
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +176,9 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param statsToKeep the collection of the column ids for the columns which stats are kept
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Collection<Integer> statsToKeep) {
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +518,40 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static Map<Integer, Long> filteredLongMap(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382325753
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT columnsToKeepStats(Set<Integer> columnsToKeepStats) {
Review Comment:
Why not simply overload `includeColumnStats()` with a version that accepts a collection of columns? I feel that will be easier to interpret as folks are already familiar with `includeColumnStats()` that currently loads stats for all columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1388756877
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
Review Comment:
Optional: I think `with only specific column stats` may be interpreted as with some types of metrics, rather than with all types of metrics but for specific columns.
I'd consider something like this but it is up to you.
```
Copies this file with column stats only for specific columns. Manifest readers can reuse file
instances; use this method to copy data with stats only for specific columns when collecting
files.
```
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
Review Comment:
Minor: `column ids` -> `column IDs` to follow the existing Javadoc in this class.
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -165,6 +170,19 @@ public ThisT includeColumnStats() {
return newRefinedScan(table, schema, context.shouldReturnColumnStats(true));
}
+ @Override
+ public ThisT includeColumnStats(Collection<String> requestedColumns) {
+ return newRefinedScan(
+ table,
+ schema,
+ context
+ .shouldReturnColumnStats(true)
Review Comment:
Supporting both `shouldReturnColumnStats()` and `columnsToKeepStats()` will only be needed if we decide to go with the utility for copying. If `null` is a valid value in `ContentFile$copyWithStats`, I think having just a set of field IDs to keep stats for would be sufficient.
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +190,25 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
Review Comment:
In my view, this piece should not use streams, closures and must do as few extra copies as possible. It is called in a tight loop and our previous profiling showed that all this cost adds up at scale.
I would suggest extending `SerializableMap` with `filteredCopyOf`.
```
public static <K, V> SerializableMap<K, V> filteredCopyOf(Map<K, V> map, Set<K> keys) {
return map == null ? null : new SerializableMap<>(map, keys);
}
private SerializableMap(Map<K, V> map, Set<K> keys) {
Map<K, V> filteredMap = Maps.newHashMapWithExpectedSize(keys.size());
for (K key : keys) {
V value = map.get(key);
if (value != null) {
filteredMap.put(key, value);
}
}
this.copiedMap = filteredMap;
}
```
Then we can define some extra helper methods in `BaseFile` to have reasonable formatting.
```
private static <K, V> Map<K, V> copyMap(Map<K, V> map, Set<K> keys) {
if (keys == null) {
return SerializableMap.copyOf(map);
} else {
return SerializableMap.filteredCopyOf(map, keys);
}
}
private static Map<Integer, ByteBuffer> copyByteBufferMap(
Map<Integer, ByteBuffer> map, Set<Integer> keys) {
return SerializableByteBufferMap.wrap(copyMap(map, keys));
}
```
I understand we are trying to be generic here but I think the performance is critical.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
Review Comment:
Minor: `a copy of this data file` -> `a copy of this file`, it is not necessary a data file anymore. We are gradually fixing the docs.
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +77,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param requestedColumns column names for which to keep the stats. If <code>null</code> then all
Review Comment:
I think we should not support null here as a valid value independently of what we will do in `ContentFile`. I would drop the doc.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
Review Comment:
Question: Should `with stats lower bounds, ...` be `with lower bounds, ...`?
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> requestedColumnIds) {
Review Comment:
I spend a bit more time on this one. I do think we either drop support for `null` as a valid value here or drop the utility to copy. It is up to you which approach to pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392370245
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,11 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
+ org.apache.iceberg:iceberg-core:
+ - code: "java.field.serialVersionUIDChanged"
Review Comment:
Checked, but even when setting the `serialVersionUID` to `-3377238354349859240L` we have a revapi failure.
Also double checked, but `serialver` generated the same id for the old and the new code on my mac.
Do resorted to the revapi change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1389225623
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -165,6 +170,19 @@ public ThisT includeColumnStats() {
return newRefinedScan(table, schema, context.shouldReturnColumnStats(true));
}
+ @Override
+ public ThisT includeColumnStats(Collection<String> requestedColumns) {
+ return newRefinedScan(
+ table,
+ schema,
+ context
+ .shouldReturnColumnStats(true)
Review Comment:
Let's talk about this.
My main arguments will be:
- Backward compatibility - no need to change the API
- No hidden meaning for specific list values (null, empty)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375314451
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT columnsToKeepStats(Set<Integer> columnsToKeepStats) {
Review Comment:
I think this should be `includeColumnStatsFor` to align with the readability conventions and match the `includeColumnStats` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375315462
##########
core/src/main/java/org/apache/iceberg/V2Metadata.java:
##########
@@ -560,6 +561,11 @@ public F copy() {
throw new UnsupportedOperationException("Cannot copy IndexedDataFile wrapper");
}
+ @Override
+ public F copyWithStats(Set<Integer> columnsToKeepStats) {
Review Comment:
I think it is debatable whether to use `String` or `Integer` here. Clearly, `Integer` is easier but why should the API expose column IDs? I suspect that this should be `String` unless there is something that makes it far more difficult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1364171669
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +176,9 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param statsToKeep the collection of the column ids for the columns which stats are kept
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Collection<Integer> statsToKeep) {
Review Comment:
nit: arg can be `columnsToKeepStats`
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +77,20 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columns column ids from the table's schema
+ * @return a new scan based on this that loads column stats for specific columns.
+ */
+ default ThisT includeColumnStats(Collection<Integer> columns) {
Review Comment:
Set is probably the better type
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -186,12 +189,23 @@ public PartitionData copy() {
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (statsToKeep == null || statsToKeep.isEmpty()) {
Review Comment:
to simplify the code, we can probably move the if-else inside the `filterColumnStats` method below. so we can always do `...copyOf(filterColumnsStats(..., columnsToKeepStats)`
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -59,6 +60,11 @@ public boolean returnColumnStats() {
return false;
}
+ @Value.Default
+ public Collection<Integer> returnColumnStatsToInclude() {
Review Comment:
I would call this `columnsToIncludeStats`. return here is not intuitive here, while it makes sense for the method above.
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -121,6 +121,10 @@ protected boolean shouldReturnColumnStats() {
return context().returnColumnStats();
}
+ protected Collection<Integer> columnStatsToInclude() {
Review Comment:
nit: columnsToIncludeStats?
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -177,4 +191,26 @@ default Long fileSequenceNumber() {
default F copy(boolean withStats) {
return withStats ? copy() : copyWithoutStats();
}
+
+ /**
+ * Copies this file (potentially with or without specific column stats). Manifest readers can
+ * reuse file instances; use this method to copy data when collecting files from tasks.
+ *
+ * @param withStats Will copy this file without file stats if set to <code>false</code>.
+ * @param statsToKeep Will keep stats only for these columns. Not used if <code>withStats</code>
+ * is set to <code>false</code>.
+ * @return a copy of this data file. If "withStats" is set to <code>false</code> the file will not
+ * contain lower bounds, upper bounds, value counts, null value counts, or nan value counts.
+ * If "withStats" is set to <code>true</code> and the "statsToKeep" is not empty then only
+ * specific column stats will be kept.
+ */
+ default F copy(boolean withStats, Collection<Integer> statsToKeep) {
Review Comment:
yeah. I think it is more important to keep the interface lean. I am wondering if we can move them into a util class like `ContentFileUtil` that is similar to `ManifestFileUtil`.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,19 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param statsToKeep the collection of the column ids for the columns which stats are kept
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithSpecificStats(Collection<Integer> statsToKeep) {
Review Comment:
nit: `columnsToKeepStats` arg name is probably more accurate. the type probably should be a `Set` to be accurate.
Regarding method name, `Specific` is not conveying much in this context. I am wondering if we can simply name it `copyWithStats(Set<Integer> columnsToKeepStats)`.
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +518,40 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static Map<Integer, Long> filteredLongMap(
Review Comment:
we probably can combine these two util method into one with generic, like
```
private static <T> Map<Integer, T> filterColumnsStats(Map<Integer, T> stats, Set<Integer> columnsToKeepStats)
```
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -125,6 +131,16 @@ TableScanContext shouldReturnColumnStats(boolean returnColumnStats) {
.build();
}
+ TableScanContext shouldReturnSpecificColumnStats(Collection<Integer> columStatsToInclude) {
Review Comment:
`should` is typically used for boolean arg. how about just `columnsToIncludeStats(Set<Integer> columns)`? similar to the `selectColumns` method below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1363537334
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -177,4 +191,26 @@ default Long fileSequenceNumber() {
default F copy(boolean withStats) {
return withStats ? copy() : copyWithoutStats();
}
+
+ /**
+ * Copies this file (potentially with or without specific column stats). Manifest readers can
+ * reuse file instances; use this method to copy data when collecting files from tasks.
+ *
+ * @param withStats Will copy this file without file stats if set to <code>false</code>.
+ * @param statsToKeep Will keep stats only for these columns. Not used if <code>withStats</code>
+ * is set to <code>false</code>.
+ * @return a copy of this data file. If "withStats" is set to <code>false</code> the file will not
+ * contain lower bounds, upper bounds, value counts, null value counts, or nan value counts.
+ * If "withStats" is set to <code>true</code> and the "statsToKeep" is not empty then only
+ * specific column stats will be kept.
+ */
+ default F copy(boolean withStats, Collection<Integer> statsToKeep) {
Review Comment:
I have found nicer to have a default method, than doing the calculation in `BaseDistributedDataScan.toFileTasks` and in `ManifestGroup.createFileScanTasks` too.
I am ok with moving this to the methods mentioned above if you think it is more important to keep the interface thin. Please leave comment, if you still think we should move the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1768776022
> > @pvary I think we probably want to push the `copyStatsForColumns` down to ManifestReader. https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestReader.java#L299
>
> That is for reading the data from the manifest file. If we want at statistics for at least one column, then the manifest file reading schema should contain the stat fields, like:
>
> ```
> private static final Set<String> STATS_COLUMNS =
> ImmutableSet.of(
> "value_counts",
> "null_value_counts",
> "nan_value_counts",
> "lower_bounds",
> "upper_bounds",
> "record_count");
> ```
>
> So we can not do filtering here. We need to read the stat fields from the manifest file, and then filter later for columns where we do not need it.
If we look at this line
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestReader.java#L299
it calls this method from `ContentFile`
```
default F copy(boolean withStats) {
return withStats ? copy() : copyWithoutStats();
}
```
if we push down the selection to the ManifestReader, it can call the new `copyWithSpecificStats` method that you added in this PR.
I understand the current code is for metadata column selection/projection, not the columns selected to include stats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682641
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1389220119
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files.
+ *
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> requestedColumnIds) {
Review Comment:
I am not fan of the special value settings, where `null`, and `empty` have specific meaning, so I ended up not defining what we are doing with `null` on the API.
Still behind the scenes in the constructors for `BaseFile`, `GenericDataFile`, etc, I need to keep the `null` as a possible value, but at least that is not a public API.
Will talk about this one more time, before finalising the solution...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1389171088
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +77,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param requestedColumns column names for which to keep the stats. If <code>null</code> then all
Review Comment:
I would guess that the situation is the similar with the other `@Nullable` `TableScanContext` attributes, like:
- snapshotId
- selectedColumns
- projectedSchema
- fromSnapshotId
- toSnapshotId
- branch
We have an undefined default behaviour which could be archived with not setting the values, or setting them to `null`. The only difference here is that we define this default behaviour. For consistency's sake we can remove the comment, but the behaviour will remain the same.
Do I miss something?
Thanks for the detailed review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1355020627
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/enumerator/TestContinuousSplitPlannerImpl.java:
##########
@@ -533,6 +533,116 @@ public void testMaxPlanningSnapshotCount() throws Exception {
thirdResult, snapshot1, snapshot2, ImmutableSet.of(dataFile2.path().toString()));
}
+ @Test
+ public void testTableScanNoStats() throws Exception {
+ appendTwoSnapshots();
+
+ ScanContext scanContext =
+ ScanContext.builder()
+ .includeColumnStats(false)
+ .startingStrategy(StreamingStartingStrategy.TABLE_SCAN_THEN_INCREMENTAL)
+ .build();
+ ContinuousSplitPlannerImpl splitPlanner =
+ new ContinuousSplitPlannerImpl(tableResource.tableLoader().clone(), scanContext, null);
+
+ ContinuousEnumerationResult initialResult = splitPlanner.planSplits(null);
+ Assert.assertEquals(1, initialResult.splits().size());
Review Comment:
it would be better to use AssertJ-style assertions for newly added code as that makes migrating away from JUnit4 easier. See also https://github.com/apache/iceberg/blob/a3aff95f9e60962240b94242e24a778760bdd1d9/CONTRIBUTING.md#assertj.
That particular check would then be `assertThat(initialResult.splits()).hasSize(1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365195690
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -177,4 +191,26 @@ default Long fileSequenceNumber() {
default F copy(boolean withStats) {
return withStats ? copy() : copyWithoutStats();
}
+
+ /**
+ * Copies this file (potentially with or without specific column stats). Manifest readers can
+ * reuse file instances; use this method to copy data when collecting files from tasks.
+ *
+ * @param withStats Will copy this file without file stats if set to <code>false</code>.
+ * @param statsToKeep Will keep stats only for these columns. Not used if <code>withStats</code>
+ * is set to <code>false</code>.
+ * @return a copy of this data file. If "withStats" is set to <code>false</code> the file will not
+ * contain lower bounds, upper bounds, value counts, null value counts, or nan value counts.
+ * If "withStats" is set to <code>true</code> and the "statsToKeep" is not empty then only
+ * specific column stats will be kept.
+ */
+ default F copy(boolean withStats, Collection<Integer> statsToKeep) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365732924
##########
api/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
Review Comment:
this class should be in the core module (not api)
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
Review Comment:
return type can be `SerializableMap`
##########
core/src/main/java/org/apache/iceberg/GenericDeleteFile.java:
##########
@@ -67,23 +69,30 @@ class GenericDeleteFile extends BaseFile<DeleteFile> implements DeleteFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param fullCopy whether to copy all fields or to drop column-level stats.
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDeleteFile(GenericDeleteFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDeleteFile(GenericDeleteFile toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
nit: statsNeeded -> columnsToKeepStats for consistency
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadOptions.java:
##########
@@ -96,6 +96,9 @@ private FlinkReadOptions() {}
public static final ConfigOption<Boolean> INCLUDE_COLUMN_STATS_OPTION =
ConfigOptions.key(PREFIX + INCLUDE_COLUMN_STATS).booleanType().defaultValue(false);
+ public static final String COLUMN_STATS_TO_KEEP = "column-stats-to-keep";
Review Comment:
columns-to-keep-stats
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
Review Comment:
nit: I know `SerializableMap` handles null properly. it doesn't really matter in practice. but I still find it more intuitive to check the map null first.
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
is it necessary to check null value?
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +176,10 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
nit: statsToKeep -> columnsToKeepStats for consistency
##########
api/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
+ * @return The copied file
+ */
+ public static <F extends ContentFile<K>, K> K copy(
+ F file, boolean withStats, Set<Integer> statsToKeep) {
Review Comment:
nit: statsToKeep -> columnsToKeepStats for consistency
##########
api/src/test/java/org/apache/iceberg/TestHelpers.java:
##########
@@ -662,6 +663,11 @@ public DataFile copyWithoutStats() {
return this;
}
+ @Override
+ public DataFile copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
nit: statsToKeep -> columnsToKeepStats for consistency
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +68,30 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param fullCopy whether to copy all fields or to drop column-level stats.
+ * @param statsToKeep a set of column ids to keep stats. If empty or <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(GenericDataFile toCopy, boolean fullCopy, Set<Integer> statsToKeep) {
Review Comment:
nit: statsNeeded -> columnsToKeepStats for consistency
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -154,6 +156,11 @@ ManifestGroup caseSensitive(boolean newCaseSensitive) {
return this;
}
+ ManifestGroup columnStatsToKeep(Set<Integer> newColumnStatsToKeep) {
+ this.columnStatsToKeep = newColumnStatsToKeep;
Review Comment:
the convention is to do a defensive copy here. see the `select` method above from Line 148
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
Review Comment:
nit: is it tiny bit simpler to use Java stream API for filter.
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -165,6 +169,12 @@ public ThisT includeColumnStats() {
return newRefinedScan(table, schema, context.shouldReturnColumnStats(true));
}
+ @Override
+ public ThisT includeColumnStats(Set<Integer> statsNeeded) {
Review Comment:
nit: statsNeeded -> columnsToIncludeStats for consistency
##########
core/src/main/java/org/apache/iceberg/V2Metadata.java:
##########
@@ -560,6 +561,11 @@ public F copy() {
throw new UnsupportedOperationException("Cannot copy IndexedDataFile wrapper");
}
+ @Override
+ public F copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
nit: statsToKeep -> columnsToKeepStats for consistency
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -125,6 +132,16 @@ TableScanContext shouldReturnColumnStats(boolean returnColumnStats) {
.build();
}
+ TableScanContext columnsToIncludeStats(Set<Integer> columnStatsToInclude) {
+ Preconditions.checkState(
+ returnColumnStats(),
+ "Cannot select column stats to include when column stats are not returned");
Review Comment:
nit: wondering if this small rewording can be a little clearer
```
Cannot select columns to keep stats when column stats are not returned
```
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -381,19 +390,22 @@ static class TaskContext {
private final DeleteFileIndex deletes;
private final ResidualEvaluator residuals;
private final boolean dropStats;
+ private final Set<Integer> statsToKeep;
private final ScanMetrics scanMetrics;
TaskContext(
PartitionSpec spec,
DeleteFileIndex deletes,
ResidualEvaluator residuals,
boolean dropStats,
+ Set<Integer> statsToKeep,
Review Comment:
nit: statsNeeded -> columnsToKeepStats for consistency
##########
docs/flink-configuration.md:
##########
@@ -130,6 +130,7 @@ env.getConfig()
| streaming | connector.iceberg.streaming | N/A | false | Sets whether the current task runs in streaming or batch mode.
|
| monitor-interval | connector.iceberg.monitor-interval | N/A | 60s | Monitor interval to discover splits from new snapshots. Applicable only for streaming read.
|
| include-column-stats | connector.iceberg.include-column-stats | N/A | false | Create a new scan from this that loads the column stats with each data file. Column stats include: value count, null value count, lower bounds, and upper bounds.
|
+| column-stats-to-keep | connector.iceberg.column-stats-to-keep | N/A | empty | Create a new scan from this that loads the column stats with each data file for the specified column ids. Column stats include: value count, null value count, lower bounds, and upper bounds.
|
Review Comment:
columns-to-keep-stats
##########
core/src/main/java/org/apache/iceberg/V1Metadata.java:
##########
@@ -485,6 +486,11 @@ public DataFile copy() {
return wrapped.copy();
}
+ @Override
+ public DataFile copyWithStats(Set<Integer> statsToKeep) {
Review Comment:
nit: statsToKeep -> columnsToKeepStats for consistency
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -190,4 +203,11 @@ public int maxAllowedPlanningFailures() {
.defaultValue(FlinkReadOptions.MAX_ALLOWED_PLANNING_FAILURES_OPTION.defaultValue())
.parse();
}
+
+ public static Set<Integer> split(String text) {
Review Comment:
this can be private
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
Review Comment:
columnsToKeepStats
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -417,6 +429,10 @@ boolean shouldKeepStats() {
return !dropStats;
}
+ Set<Integer> statsToKeep() {
Review Comment:
nit: statsNeeded -> columnsToKeepStats for consistency
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java:
##########
@@ -464,6 +476,11 @@ public Builder includeColumnStats(boolean newIncludeColumnStats) {
return this;
}
+ public Builder columnStatsToKeep(Set<Integer> newColumnStatsToKeep) {
Review Comment:
nit: columnsToKeepStats
we also need to expose this method in `IcebergSource`
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
+ return split(
Review Comment:
I am also wondering if it is intuitive to ask users to configure this via field ids. Users tend to interact with tables via field names (not ids)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382331393
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
I find this logic a bit hard to follow cause we control the behavior using two variables. Any way we can simplify that? We could have something like below.
```
Scan.java
includeColumnStats() -> include all stats
includeColumnStats(Collection<String> requestedColumns) -> include only stats for requested columns
ContentFile.java
copy() -> copy preserving all loaded stats (existing)
copyWithoutStats() -> copy without any loaded stats (existing)
copy(boolean withStats) -> calls either copy() or copyWithoutStats() (existing)
copyWithStats(Collection<String> requestedColumns) -> copy with stats only for requested columns
```
We can then maintain a list of columns to project stats in our scans or in the scan context object.
- A collection with a single element `*` means we will call `copy()` on files.
- Null or empty collection means we will call `copyWithoutStats()` on files.
- A collection with column names means we will call `copyWithStats(requestedColumns)` on files.
All of the logic above can be incapsulated in a single copy method in our base scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382331518
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
Any thoughts, @pvary @rdblue @stevenzwu?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1370442734
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files. If the
+ * columnsToKeepStats set is empty or <code>null</code>, then all column stats will be kept.
+ *
+ * @param columnsToKeepStats the set of the column ids for the columns which stats are kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> columnsToKeepStats) {
+ throw new UnsupportedOperationException(
+ this.getClass().getName() + " doesn't implement copyWithSpecificStats");
Review Comment:
copyWithSpecificStats needs to be updated to `copyWithStats(Set<Integer>)`
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -174,8 +177,10 @@ public PartitionData copy() {
*
* @param toCopy a generic data file to copy.
* @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
*/
- BaseFile(BaseFile<F> toCopy, boolean fullCopy) {
+ BaseFile(BaseFile<F> toCopy, boolean fullCopy, Set<Integer> columnsToKeepStats) {
Review Comment:
the arg name `fullCopy` is not accurate anymore. maybe rename it to `copyStats`?
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
Review Comment:
I meant `columnIds.stream().map(columnId -> map.get(columnId))...`
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
Review Comment:
got it. this is good then
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
Review Comment:
ok. I am fine with it.
It is either check the map or the columnIds first. not sure about extra null check.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files. If the
Review Comment:
should `If the columnsToKeepStats set is empty ...` be put in the `@param` below?
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
If there are null values in lower_bound or upper_bound, why do we want to filter them out? shouldn't we keep the same behavior when copying stats for selected columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365196518
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -121,6 +121,10 @@ protected boolean shouldReturnColumnStats() {
return context().returnColumnStats();
}
+ protected Collection<Integer> columnStatsToInclude() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369761574
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -417,6 +429,10 @@ boolean shouldKeepStats() {
return !dropStats;
}
+ Set<Integer> statsToKeep() {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -125,6 +132,16 @@ TableScanContext shouldReturnColumnStats(boolean returnColumnStats) {
.build();
}
+ TableScanContext columnsToIncludeStats(Set<Integer> columnStatsToInclude) {
+ Preconditions.checkState(
+ returnColumnStats(),
+ "Cannot select column stats to include when column stats are not returned");
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375354727
##########
core/src/main/java/org/apache/iceberg/V2Metadata.java:
##########
@@ -560,6 +561,11 @@ public F copy() {
throw new UnsupportedOperationException("Cannot copy IndexedDataFile wrapper");
}
+ @Override
+ public F copyWithStats(Set<Integer> columnsToKeepStats) {
Review Comment:
Is there an easy way to convert column name to column id in case of schema evolution? How should we handle cases where the column is renamed, or a column is dropped and a new column is created with the same name?
Would it be enough to use the `Schema.findField` for this, and accept that we can only enable the feature for current columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392375838
##########
core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java:
##########
@@ -102,7 +102,8 @@ public CloseableIterable<FileScanTask> planFiles() {
snapshotIds.contains(manifestEntry.snapshotId())
&& manifestEntry.status() == ManifestEntry.Status.ADDED)
.specsById(table().specs())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392370573
##########
core/src/main/java/org/apache/iceberg/BaseDistributedDataScan.java:
##########
@@ -368,7 +369,9 @@ private CloseableIterable<ScanTask> toFileTasks(
ScanMetricsUtil.fileTask(scanMetrics(), dataFile, deleteFiles);
return new BaseFileScanTask(
- copyDataFiles ? dataFile.copy(shouldReturnColumnStats()) : dataFile,
+ copyDataFiles
Review Comment:
Created the method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1812123613
Thanks @nastra, @rdblue, @stevenzwu, @aokolnychyi for the diligent reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369758919
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
Review Comment:
I expect low number of `columnIds` and high number of columns. So I think this is the fastest way to filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382659616
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
+1 on using string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1367842806
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
+ return split(
Review Comment:
Using column names opens an entirely new can of worms. How do we handle schema evolution? If we accept, that we do not solve this issue, then we can use the `Schema.findField` for easy conversion, but then the users could not solve the issue themselves if they used schema evolution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369755601
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
Review Comment:
I would keep this way, to avoid the extra null check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382325507
##########
api/src/main/java/org/apache/iceberg/Scan.java:
##########
@@ -77,6 +78,21 @@ public interface Scan<ThisT, T extends ScanTask, G extends ScanTaskGroup<T>> {
*/
ThisT includeColumnStats();
+ /**
+ * Create a new scan from this that loads the column stats for the specific columns with each data
+ * file. If the columns set is empty or <code>null</code> then all column stats will be kept, if
+ * {@link #includeColumnStats()} is set.
+ *
+ * <p>Column stats include: value count, null value count, lower bounds, and upper bounds.
+ *
+ * @param columnsToKeepStats column ids from the table's schema
Review Comment:
+1.
We have `select` that accepts `Collection<String> columns` below. I understand using IDs is straightforward but someone has to do the conversion from name to ID and we better do it ourselves behind this API. If an external long-running process needs to ensure the ID did not change, it can it do it outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1811313831
I went through Ryan's comments one more time. They seem to be addressed. I also think the current version is simpler. Let's merge it as is and follow up if needed to unblock subsequent changes in Flink.
@rdblue, please let us know if you spot anything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382331393
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
I find this logic a bit hard to follow cause we control the behavior using two variables. Any way we can simplify that? We could have something like below.
```
Scan.java
includeColumnStats() -> include all stats
includeColumnStats(Collection<String> requestedColumns) -> include only stats for requested columns
ContentFile.java
copy() -> copy preserving all loaded stats (existing)
copyWithoutStats() -> copy without any loaded stats (existing)
copy(boolean withStats) -> calls either copy() or copyWithoutStats() (existing)
copyWithStats(Collection<String> requestedColumns) -> copy with stats only for requested columns
```
We can then maintain a list of columns to project stats in our scans or in the scan context object.
- A collection with a single element `*` means we will call copy() on files.
- Null or empty collection means we will call copyWithoutStats() on files.
- A collection with column names means we will call copyWithStats() on files.
All of that logic above can be incapsulated in a single copy method in our base scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1367939105
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
+ return split(
Review Comment:
Regarding schema evolution (like field rename), users have to adjust the name reference. this is also applicable to filter expressions or column projections. Users typically reference fields by name.
In another thought, I am also not sure we need to expose this to users directly. The only usage of column stats is the watermark alignment. we will need to expose `enableWatermarkAlignment(String watermarkField)` anyway. there is probably no need to expose this directly, at least not yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369754161
##########
api/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
Review Comment:
Yeah, you are right.
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375163433
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
thanks. double fetching is probably ok since the columnsToKeepStats is likely not big and Map get is a O(1) anyway.
The alternative is to start a discussion and revisit the spec or Parquet implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375354727
##########
core/src/main/java/org/apache/iceberg/V2Metadata.java:
##########
@@ -560,6 +561,11 @@ public F copy() {
throw new UnsupportedOperationException("Cannot copy IndexedDataFile wrapper");
}
+ @Override
+ public F copyWithStats(Set<Integer> columnsToKeepStats) {
Review Comment:
@rdblue: Is there an easy way to convert column name to column id in case of schema evolution? How should we handle cases where the column is renamed, or a column is dropped and a new column is created with the same name?
Would it be enough to use the `Schema.findField` for this, and accept that we can only enable the feature for current columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1374109329
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
Changed to ensure that even `null` values are copied if we accept the cost of possible double fetching the entries from the map.
Still not sure that this is necessary, but I will be on PTO next week, so wanted to have it here as a possibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1372568261
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
sure. but could lower_bound and upper_bound have null value? how do we distinguish btw these two scenario?
e.g. here is the description for lower_bound
```
Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1367174997
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java:
##########
@@ -464,6 +476,11 @@ public Builder includeColumnStats(boolean newIncludeColumnStats) {
return this;
}
+ public Builder columnStatsToKeep(Set<Integer> newColumnStatsToKeep) {
Review Comment:
nit: columnsToKeepStats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369764771
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -152,6 +155,16 @@ public boolean includeColumnStats() {
.parse();
}
+ public Set<Integer> columnStatsToKeep() {
+ return split(
Review Comment:
Might need stats for previous versions of columns (old files contain old stats for old columns).
Removed for now anyway
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java:
##########
@@ -464,6 +476,11 @@ public Builder includeColumnStats(boolean newIncludeColumnStats) {
return this;
}
+ public Builder columnStatsToKeep(Set<Integer> newColumnStatsToKeep) {
Review Comment:
Fixed everywhere (I hope, will check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369760806
##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -165,6 +169,12 @@ public ThisT includeColumnStats() {
return newRefinedScan(table, schema, context.shouldReturnColumnStats(true));
}
+ @Override
+ public ThisT includeColumnStats(Set<Integer> statsNeeded) {
Review Comment:
Fixed everywhere (I hope, will check).
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -154,6 +156,11 @@ ManifestGroup caseSensitive(boolean newCaseSensitive) {
return this;
}
+ ManifestGroup columnStatsToKeep(Set<Integer> newColumnStatsToKeep) {
+ this.columnStatsToKeep = newColumnStatsToKeep;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1355022285
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -191,6 +192,11 @@ public DataFile copy() {
throw new UnsupportedOperationException("Not implemented: copy");
}
+ @Override
+ public DataFile copyWithSpecificStats(Collection<Integer> statsToKeep) {
Review Comment:
I don't think this is needed when there's a default impl in the interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1375315526
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
This should not keep all stats when empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1373440573
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
thanks for the unit test code. I am still not comfortable of making the assumption as the Iceberg spec clearly said null stats value is possible. what if Trino parquet writer produces min/max stats with null value.
```
Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]
```
As least, there is a mismatch in the Parquet writer implementation and the spec. Unless we are going to clarify/update the spec, I feel it is safer to do `map.containsKey(key)`.
Maybe @RussellSpitzer and @rdblue can chime in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369760407
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
Yes, I found this in some tests, that the map could contain null values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369763072
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadOptions.java:
##########
@@ -96,6 +96,9 @@ private FlinkReadOptions() {}
public static final ConfigOption<Boolean> INCLUDE_COLUMN_STATS_OPTION =
ConfigOptions.key(PREFIX + INCLUDE_COLUMN_STATS).booleanType().defaultValue(false);
+ public static final String COLUMN_STATS_TO_KEEP = "column-stats-to-keep";
Review Comment:
Removed, as per another comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369762755
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadConf.java:
##########
@@ -190,4 +203,11 @@ public int maxAllowedPlanningFailures() {
.defaultValue(FlinkReadOptions.MAX_ALLOWED_PLANNING_FAILURES_OPTION.defaultValue())
.parse();
}
+
+ public static Set<Integer> split(String text) {
Review Comment:
Removed as per another comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1369756713
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
Review Comment:
I would need to change the `SerializableByteBufferMap` for this, so I kept as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382331393
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
I find this logic a bit hard to follow cause we control the behavior using two variables. Any way we can simplify that? We could have something like below.
```
Scan.java
includeColumnStats() -> include all stats
includeColumnStats(Collection<String> requestedColumns) -> include only stats for requested columns
ContentFile.java
copy() -> copy preserving all loaded stats (existing)
copyWithoutStats() -> copy without any loaded stats (existing)
copy(boolean withStats) -> calls either copy() or copyWithoutStats() (existing)
copyWithStats(Collection<String> requestedColumns) -> copy with stats only for requested columns
```
We can then maintain a list of columns to project stats in our scans or in the scan context object.
- A collection with a single element `*` means we will call `copy()` on files.
- Null or empty collection means we will call `copyWithoutStats()` on files.
- A collection with column names means we will call `copyWithStats(requestedColumns)` on files.
All of that logic above can be incapsulated in a single copy method in our base scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371948395
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
Review Comment:
This would be the identical one:
```
Map<Integer, TypeT> filtered =
columnIds.stream()
.filter(id -> map.get(id) != null)
.collect(Collectors.toMap(Function.identity(), map::get));
```
But this calls `map.get(id)` twice, if no compiler optimization kicks in.
That is why I kept the original code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1373440573
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
thanks for the unit test code. I am still not comfortable of making the assumption as the Iceberg spec clearly said null stats value is possible.
```
Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]
```
Unless we are going to clarify/update the spec, I feel it is safer to do `map.containsKey(key)`.
Maybe @RussellSpitzer and @rdblue can chime in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371954933
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files. If the
+ * columnsToKeepStats set is empty or <code>null</code>, then all column stats will be kept.
+ *
+ * @param columnsToKeepStats the set of the column ids for the columns which stats are kept.
+ * @return a copy of this data file, with stats lower bounds, upper bounds, value counts, null
+ * value counts, and nan value counts for only specific columns.
+ */
+ default F copyWithStats(Set<Integer> columnsToKeepStats) {
+ throw new UnsupportedOperationException(
+ this.getClass().getName() + " doesn't implement copyWithSpecificStats");
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371953259
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -504,6 +508,27 @@ private static Map<Integer, ByteBuffer> toReadableByteBufferMap(Map<Integer, Byt
}
}
+ private static <TypeT> Map<Integer, TypeT> filterColumnsStats(
+ Map<Integer, TypeT> map, Set<Integer> columnIds) {
+ if (columnIds == null || columnIds.isEmpty()) {
+ return SerializableMap.copyOf(map);
+ }
+
+ if (map == null) {
+ return null;
+ }
+
+ Map<Integer, TypeT> filtered = Maps.newHashMapWithExpectedSize(columnIds.size());
+ for (Integer columnId : columnIds) {
+ TypeT value = map.get(columnId);
+ if (value != null) {
Review Comment:
My previous answer was not clear enough. Let me rephrase: "we can have maps where the columnId is not contained in the key list". In this case the `map.get(columnId);` returns `null`. To return the correct map, we should not add the key to the list in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371956676
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -165,6 +166,20 @@ default Long fileSequenceNumber() {
*/
F copyWithoutStats();
+ /**
+ * Copies this file with only specific column stats. Manifest readers can reuse file instances;
+ * use this method to copy data and only copy specific stats when collecting files. If the
Review Comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383715528
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+
+public class ContentFileUtil {
+ private ContentFileUtil() {}
+
+ /**
+ * Copies the {@link ContentFile} with the specific stat settings.
+ *
+ * @param file a generic data file to copy.
+ * @param withStats whether to keep any stats
+ * @param columnsToKeepStats a set of column ids to keep stats. If empty or <code>null</code> then
+ * every column stat is kept.
Review Comment:
I converted the `Scan` to use the column names instead of the column ids, like: `Scan.includeColumnStats(Collection<String> requestedColumns)`
For the `ContentFile` we do not have the `Schema` at hand, so I had to stick to the `ContentFile.copyWithStats(Set<Integer> requestedColumnIds)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1393295105
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,11 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
+ org.apache.iceberg:iceberg-core:
+ - code: "java.field.serialVersionUIDChanged"
Review Comment:
Yeah, I am not sure what revapi actually does. I doubt they compare actual values. I think we should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #8803:
URL: https://github.com/apache/iceberg/pull/8803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1391932246
##########
core/src/main/java/org/apache/iceberg/BaseDistributedDataScan.java:
##########
@@ -368,7 +369,9 @@ private CloseableIterable<ScanTask> toFileTasks(
ScanMetricsUtil.fileTask(scanMetrics(), dataFile, deleteFiles);
return new BaseFileScanTask(
- copyDataFiles ? dataFile.copy(shouldReturnColumnStats()) : dataFile,
+ copyDataFiles
Review Comment:
Optional: I am not a big fan of using ternary operators when they are split across multiple lines. To me, this makes the logic harder to follow. I'd consider adding a method like below to `BaseScan`.
```
protected <F extends ContentFile<F>> F copy(F file) {
return ContentFileUtil.copy(file, shouldReturnColumnStats(), columnsToKeepStats());
}
```
This will make the invocation shorter and we may reuse this method in other scans in the future.
##########
core/src/main/java/org/apache/iceberg/DataScan.java:
##########
@@ -55,7 +55,8 @@ protected ManifestGroup newManifestGroup(
.filterData(filter())
.specsById(table().specs())
.scanMetrics(scanMetrics())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Here as well.
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,11 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
+ org.apache.iceberg:iceberg-core:
+ - code: "java.field.serialVersionUIDChanged"
Review Comment:
While I think it should be fine, here is an idea. Java comes with `serialver` utility that allows us to generate the version UID prior to the change in this PR. We can use that value instead of `1L` to be fully compatible. We don't modify the serialization of this class, we just missed to assign `serialVersionUID`. If we can recover the default value, we shouldn't worry about compatibility.
Here is the value I I got locally:
```
cd core/build/classes/java/main
serialver org.apache.iceberg.util.SerializableMap
org.apache.iceberg.util.SerializableMap: private static final long serialVersionUID = -3377238354349859240L;
```
Could you double check, @pvary? If not, we can keep it as is.
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalAppendScan.java:
##########
@@ -83,7 +83,8 @@ private CloseableIterable<FileScanTask> appendFilesFromSnapshots(List<Snapshot>
snapshotIds.contains(manifestEntry.snapshotId())
&& manifestEntry.status() == ManifestEntry.Status.ADDED)
.specsById(table().specs())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
I think you have defined `columnsToKeepStats()` method, which you can call directly now.
See `BaseScan`.
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +67,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> requestedColumnIds) {
+ super(toCopy, copyStats, requestedColumnIds);
}
/** Constructor for Java serialization. */
GenericDataFile() {}
@Override
public DataFile copyWithoutStats() {
- return new GenericDataFile(this, false /* drop stats */);
+ return new GenericDataFile(this, false /* drop stats */, null);
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> requestedColumnIds) {
+ return new GenericDataFile(this, true, requestedColumnIds);
}
@Override
public DataFile copy() {
- return new GenericDataFile(this, true /* full copy */);
+ return new GenericDataFile(this, true /* full copy */, null);
Review Comment:
You may consider overloading the constructor so that you don't have to pass an extra null here or adding the comment for the second argument (we have a comment for `true` but not `null`).
##########
core/src/main/java/org/apache/iceberg/GenericDeleteFile.java:
##########
@@ -67,23 +68,31 @@ class GenericDeleteFile extends BaseFile<DeleteFile> implements DeleteFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDeleteFile(GenericDeleteFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDeleteFile(
+ GenericDeleteFile toCopy, boolean copyStats, Set<Integer> requestedColumnIds) {
+ super(toCopy, copyStats, requestedColumnIds);
}
/** Constructor for Java serialization. */
GenericDeleteFile() {}
@Override
public DeleteFile copyWithoutStats() {
- return new GenericDeleteFile(this, false /* drop stats */);
+ return new GenericDeleteFile(this, false /* drop stats */, null);
+ }
+
+ @Override
+ public DeleteFile copyWithStats(Set<Integer> requestedColumnIds) {
+ return new GenericDeleteFile(this, true, requestedColumnIds);
}
@Override
public DeleteFile copy() {
- return new GenericDeleteFile(this, true /* full copy */);
+ return new GenericDeleteFile(this, true /* full copy */, null);
Review Comment:
Same as in `GenericDataFile`.
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -78,7 +78,8 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.select(scanColumns())
.filterData(filter())
.filterManifestEntries(entry -> changelogSnapshotIds.contains(entry.snapshotId()))
- .ignoreExisting();
+ .ignoreExisting()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Same here.
##########
core/src/main/java/org/apache/iceberg/BaseFile.java:
##########
@@ -185,13 +188,30 @@ public PartitionData copy() {
this.partitionType = toCopy.partitionType;
this.recordCount = toCopy.recordCount;
this.fileSizeInBytes = toCopy.fileSizeInBytes;
- if (fullCopy) {
- this.columnSizes = SerializableMap.copyOf(toCopy.columnSizes);
- this.valueCounts = SerializableMap.copyOf(toCopy.valueCounts);
- this.nullValueCounts = SerializableMap.copyOf(toCopy.nullValueCounts);
- this.nanValueCounts = SerializableMap.copyOf(toCopy.nanValueCounts);
- this.lowerBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.lowerBounds));
- this.upperBounds = SerializableByteBufferMap.wrap(SerializableMap.copyOf(toCopy.upperBounds));
+ if (copyStats) {
+ if (requestedColumnIds == null) {
Review Comment:
Optional: I'd consider adding a few helper methods to simplify this block. We will have 3 quite big branches now. I understand we will do more null checks but I doubt it will have any noticeable performance impact.
```
private static <K, V> Map<K, V> copyMap(Map<K, V> map, Set<K> keys) {
return keys == null ? SerializableMap.copyOf(map) : SerializableMap.filteredCopyOf(map, keys);
}
private static Map<Integer, ByteBuffer> copyByteBufferMap(
Map<Integer, ByteBuffer> map, Set<Integer> keys) {
return SerializableByteBufferMap.wrap(copyMap(map, keys));
}
...
if (copyStats) {
this.columnSizes = copyMap(toCopy.columnSizes, requestedColumnIds);
this.valueCounts = copyMap(toCopy.valueCounts, requestedColumnIds);
this.nullValueCounts = copyMap(toCopy.nullValueCounts, requestedColumnIds);
this.nanValueCounts = copyMap(toCopy.nanValueCounts, requestedColumnIds);
this.lowerBounds = copyByteBufferMap(toCopy.lowerBounds, requestedColumnIds);
this.upperBounds = copyByteBufferMap(toCopy.upperBounds, requestedColumnIds);
}
```
Up to you here, @pvary. Keep it as is if you want.
##########
core/src/main/java/org/apache/iceberg/DataTableScan.java:
##########
@@ -76,7 +76,8 @@ public CloseableIterable<FileScanTask> doPlanFiles() {
.filterData(filter())
.specsById(table().specs())
.scanMetrics(scanMetrics())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
Here too.
##########
core/src/main/java/org/apache/iceberg/ManifestGroup.java:
##########
@@ -154,6 +156,12 @@ ManifestGroup caseSensitive(boolean newCaseSensitive) {
return this;
}
+ ManifestGroup columnsToKeepStats(Set<Integer> newColumnsToKeepStats) {
+ this.columnsToKeepStats =
+ newColumnsToKeepStats == null ? null : Sets.newHashSet(newColumnsToKeepStats);
Review Comment:
This copy seems redundant but up to you.
##########
core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java:
##########
@@ -102,7 +102,8 @@ public CloseableIterable<FileScanTask> planFiles() {
snapshotIds.contains(manifestEntry.snapshotId())
&& manifestEntry.status() == ManifestEntry.Status.ADDED)
.specsById(table().specs())
- .ignoreDeleted();
+ .ignoreDeleted()
+ .columnsToKeepStats(context().columnsToKeepStats());
Review Comment:
`context().columnsToKeepStats()` -> `columnsToKeepStats()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1392385312
##########
core/src/main/java/org/apache/iceberg/GenericDataFile.java:
##########
@@ -66,23 +67,31 @@ class GenericDataFile extends BaseFile<DataFile> implements DataFile {
* Copy constructor.
*
* @param toCopy a generic data file to copy.
- * @param fullCopy whether to copy all fields or to drop column-level stats
+ * @param copyStats whether to copy all fields or to drop column-level stats.
+ * @param requestedColumnIds column ids for which to keep stats. If <code>null</code> then every
+ * column stat is kept.
*/
- private GenericDataFile(GenericDataFile toCopy, boolean fullCopy) {
- super(toCopy, fullCopy);
+ private GenericDataFile(
+ GenericDataFile toCopy, boolean copyStats, Set<Integer> requestedColumnIds) {
+ super(toCopy, copyStats, requestedColumnIds);
}
/** Constructor for Java serialization. */
GenericDataFile() {}
@Override
public DataFile copyWithoutStats() {
- return new GenericDataFile(this, false /* drop stats */);
+ return new GenericDataFile(this, false /* drop stats */, null);
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> requestedColumnIds) {
+ return new GenericDataFile(this, true, requestedColumnIds);
}
@Override
public DataFile copy() {
- return new GenericDataFile(this, true /* full copy */);
+ return new GenericDataFile(this, true /* full copy */, null);
Review Comment:
I think this usage is straightforward, and adding a new constructor would not help too much.
So I did not apply this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Enable column statistics filtering after planning [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8803:
URL: https://github.com/apache/iceberg/pull/8803#discussion_r1355016941
##########
.palantir/revapi.yml:
##########
@@ -866,6 +866,15 @@ acceptedBreaks:
old: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
new: "method void org.apache.iceberg.encryption.Ciphers::<init>()"
justification: "Static utility class - should not have public constructor"
+ "1.4.0":
Review Comment:
we usually try to avoid adding API-breaking changes by introducing a default implementation as described in https://github.com/apache/iceberg/blob/a3aff95f9e60962240b94242e24a778760bdd1d9/CONTRIBUTING.md#adding-a-default-implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org