You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rdblue (via GitHub)" <gi...@apache.org> on 2026/07/10 21:46:27 UTC
[PR] Core: Refactor ContentStats and FieldStats [iceberg]
rdblue opened a new pull request, #17159:
URL: https://github.com/apache/iceberg/pull/17159
This updates the content stats API and implementations used for columnar stats in v4.
API changes:
* Removes `StructLike` used for serialization from the interfaces
* Adds `copy` methods to support copying `TrackedFile` with stats
* Aligns interface conventions (use `Iterable`, expose full struct type as `type`, etc.)
`StatsUtil` changes:
* Reduce complexity of ID methods, reduce number of constants
* Remove `FieldStatistic` enum; not all offsets were represented as `FieldStatistic` and it also embedded util methods. All utils are now in `StatsUtil`.
* Avoid double `switch` statement to resolve position to enum and enum to field in `FieldStatsStruct` `get` and `set`
* Simplified field stats struct generation
* Replaced content stats schema visitor with a loop
* Added stats schema generation for read path based on field IDs
* Added stats schema generation for write path based on `MetricsConfig`
`ContentStats` implementation changes:
* Rename to `ContentStatsStruct` to align with other v4 types
* Implement `StructLike` methods without copying data to allow object reuse, do not rebuild field stats
* Use `setCustomType` instead of copying via builder in `set`
* Keep field stats in a map rather than syncing between list and map
* Remove unnecessary builder
* Fix raw types suppressions
`FieldStats` implementation changes:
* Rename to `FieldStatsStruct` to align with other v4 types
* Remove `SupportsIndexProjection` that is bypassed
* Implement `get` and `set` projection directly based on offset, do not rely on mapping passed to constructor
* Use internal get/set to avoid duplicated `javaClass.cast`
* Remove `equals` and `hashCode`
* Remove unnecessary builder
Test changes:
* Separated `StatsUtil` test values from production constants
* Aligned implementation tests with other v4 object tests (test accessors, get, set, projection, serialization, and copy)
* Updated tests that depend on stats classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562189367
##########
core/src/test/java/org/apache/iceberg/TestTrackedFile.java:
##########
@@ -32,7 +33,7 @@ public class TestTrackedFile {
optional(1, "id", Types.IntegerType.get()), optional(2, "data", Types.StringType.get()));
private static final Types.StructType CONTENT_STATS_TYPE =
- StatsUtil.contentStatsFor(TABLE_SCHEMA).type().asStructType();
+ StatsUtil.statsReadSchema(TABLE_SCHEMA, ImmutableList.of(1, 2));
Review Comment:
The read schema method is easiest to call for testing since it creates full stats structs for each field ID. The write schema method will omit lower and upper bounds for `MetricsMode.Counts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598756731
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -531,8 +534,11 @@ static Map<Integer, ByteBuffer> lowerBounds(ContentStats stats) {
Map<Integer, ByteBuffer> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
- if (fs != null && fs.lowerBound() != null && fs.type() != null) {
- result.put(fs.fieldId(), Conversions.toByteBuffer(fs.type(), fs.lowerBound()));
+ if (fs != null) {
+ Type boundType = fs.type().fieldType("lower_bound");
+ if (fs.lowerBound() != null && boundType != null) {
+ result.put(fs.fieldId(), Conversions.toByteBuffer(boundType, fs.lowerBound()));
Review Comment:
MetricsUtil conversions are placeholders. This was already a problem. It was just hidden before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598663963
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
+ private static final Schema SCHEMA =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.StringType.get()),
+ optional(3, "score", Types.DoubleType.get()));
+
+ private static final Types.StructType CONTENT_STATS_STRUCT =
+ StatsUtil.statsReadSchema(SCHEMA, ImmutableList.of(1, 2, 3));
+ private static final Types.StructType UNKNOWN_FIELD_STATS_STRUCT =
+ StatsUtil.fieldStatsStruct(false, Types.IntegerType.get(), 10_800, MetricsModes.Full.get());
+
+ private static final FieldStats<Long> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("id").type().asStructType(), 0L, 25L, true, 26L, 0L, 0L, null);
+ private static final FieldStats<String> DATA_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("data").type().asStructType(),
+ "a",
+ "z",
+ true,
+ 26L,
+ 0L,
+ 0L,
+ null);
+
+ @Test
+ public void testEmptyContentStats() {
+ ContentStats stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.statsFor(1)).isNull();
+ assertThat(stats.statsFor(2)).isNull();
+ assertThat(stats.statsFor(3)).isNull();
+ assertThat(stats.statsFor(4)).as("Should ignore unknown field IDs").isNull();
+ }
+
+ @Test
+ public void testSetStats() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.statsFor(1)).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(2)).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(3)).isNull();
+ }
+
+ @Test
+ public void testSetStatsWrongId() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThatThrownBy(() -> stats.setStats(2, ID_STATS))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Mismatched field stats for ID 2: actual ID 1");
+ }
+
+ @Test
+ public void testSetStatsUnknownField() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ FieldStats<Integer> fieldStats =
+ new FieldStatsStruct<>(UNKNOWN_FIELD_STATS_STRUCT, 0, 10, false, 8, 3, 0, null);
+
+ assertThatThrownBy(() -> stats.setStats(4, fieldStats))
+ .hasMessage("Cannot set stats for unknown field ID: 4")
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
+ public void testGetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType contentStatsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(contentStatsStruct);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.get(0, FieldStats.class)).isEqualTo(DATA_STATS);
+ assertThat(stats.get(1, FieldStats.class)).isNull();
+ assertThat(stats.get(2, FieldStats.class)).isEqualTo(ID_STATS);
+ }
+
+ @Test
+ public void testSetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType statsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(statsStruct);
+
+ stats.set(1, null);
+ stats.set(2, ID_STATS);
+ stats.set(0, DATA_STATS);
+
+ assertThat(stats.statsFor(SCHEMA.findField("id").fieldId())).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("data").fieldId())).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("score").fieldId())).isNull();
+ }
+
+ @Test
+ public void testSize() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.size()).isEqualTo(3);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ ContentStats copy = stats.copy();
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(stats.type());
+
+ // each field stats is deep-copied using its own copy() method
+ assertThat(copy.statsFor(1)).isSameAs(idStatsCopy);
+ assertThat(copy.statsFor(2)).isSameAs(dataStatsCopy);
+ assertThat(copy.statsFor(3)).isNull();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testFilteredCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ // copy just the stats for field ID 2
+ ContentStats copy = stats.copy(ImmutableSet.of(2));
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(TypeUtil.select(stats.type(), ImmutableSet.of(1)));
+
Review Comment:
Yes, this is correct. This should not have passed the field IDs in both cases.
I've improved the test by creating the expected struct type directly instead of using `select` like the code did (which is why both places had the same error). It was bad testing and a bug!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598606955
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -18,13 +18,13 @@
*/
package org.apache.iceberg;
-import java.util.List;
+import java.util.Set;
import org.apache.iceberg.types.Types;
-interface ContentStats extends StructLike {
+interface ContentStats {
/** A list of all the {@link FieldStats} */
- List<FieldStats<?>> fieldStats();
+ Iterable<FieldStats<?>> fieldStats();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3581993822
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
Review Comment:
I added a comment about `MetricsUtil` conversion, but it was in tests so it wasn't obvious. I don't think that we actually want to convert between the two representations and that it was premature to add that logic.
Conversion is expensive. It allocates lots of buffers and copies data into them, but we don't really know what is going to be accessed. I think we should have a simple `Map` implementation that translates when values are accessed, or just return `null` for the old APIs.
I think we're seeing another place where you get in trouble when production code is introduced just for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562228207
##########
core/src/main/java/org/apache/iceberg/TrackedFileStruct.java:
##########
@@ -146,14 +146,7 @@ private TrackedFileStruct(TrackedFileStruct toCopy, boolean withStats, Set<Integ
this.tracking = toCopy.tracking != null ? toCopy.tracking.copy() : null;
this.sortOrderId = toCopy.sortOrderId;
this.deletionVector = toCopy.deletionVector != null ? toCopy.deletionVector.copy() : null;
-
- if (withStats && toCopy.contentStats != null) {
- ContentStats filtered = BaseContentStats.buildFrom(toCopy.contentStats, statsIds).build();
- this.contentStats = filtered.fieldStats().isEmpty() ? null : filtered;
- } else {
- this.contentStats = null;
- }
-
+ this.contentStats = toCopy.contentStats != null ? toCopy.contentStats.copy() : null;
Review Comment:
Looks like this breaks tests because it doesn't respect `withStats` or `statsIds` yet. I'll update it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3599247906
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
Review Comment:
This is intentional. The stats struct works like other `InternalData` classes, where the reader and writer are based on the field schema. By using the correct reader and writer, we get this check when building the reader or writer tree and don't need to check it for every value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598737339
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -516,8 +516,11 @@ static Map<Integer, Long> nanValueCounts(ContentStats stats) {
Map<Integer, Long> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
- if (fs != null && fs.nanValueCount() != null) {
- result.put(fs.fieldId(), fs.nanValueCount());
+ if (fs != null) {
+ Type boundType = fs.type().fieldType("lower_bound");
+ if (boundType.typeId() == Type.TypeID.FLOAT || boundType.typeId() == Type.TypeID.DOUBLE) {
Review Comment:
Yes, this is throw-away code that will likely not be kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3604343376
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, statsIds(fieldIds)) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ FieldStats<?> fieldToCopy = toCopy.idToFieldStats.get(fieldId);
+ if (fieldToCopy != null) {
+ idToFieldStats.put(fieldId, fieldToCopy.copy());
+ }
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
Review Comment:
```suggestion
public <T> FieldStats<T> statsFor(int fieldId) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562193487
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -76,6 +77,32 @@ class TestTrackedFileAdapters {
private static final int DELETION_VECTOR_ORDINAL =
ordinalOf(TRACKED_FILE_SCHEMA, "deletion_vector");
+ private static final Schema TABLE_SCHEMA =
+ new Schema(
+ optional(1, "id", Types.IntegerType.get()), optional(2, "score", Types.FloatType.get()));
+ private static final Types.StructType CONTENT_STATS_TYPE =
+ StatsUtil.statsReadSchema(TABLE_SCHEMA, ImmutableList.of(1, 2));
+ private static final FieldStats<Integer> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_TYPE.fieldType("id").asStructType(), 1, 1000, true, 100L, 5L, 0L, null);
+ private static final FieldStats<Float> SCORE_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_TYPE.fieldType("score").asStructType(),
+ 1.0f,
+ 100.0f,
+ true,
+ 100L,
+ 10L,
+ 3L,
+ null);
+ private static final ContentStatsStruct CONTENT_STATS =
Review Comment:
I left this in to continue testing the `MetricsUtil` conversion, but I suspect that those conversions will be done lazily.
Those are the only place where `fieldId` is needed so I think we can avoid needing to recover field ID from base ID if we rewrite them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3581671409
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -46,17 +44,20 @@ interface FieldStats<T> extends StructLike {
boolean tightBounds();
/** The total value count, including null and NaN */
- Long valueCount();
+ long valueCount();
/** The total null value count */
- Long nullValueCount();
+ long nullValueCount();
/** The total NaN value count */
- Long nanValueCount();
+ long nanValueCount();
Review Comment:
I think these accessors should reflect the API that we want for building on top of stats. So the decision should be made when we introduce classes that consume these.
I think that primitives are more locked down. They will throw a `NullPointerException` if called when there is no value, which is good for catching calls that should not be made. For example, `IS NaN` and `IS NOT NaN` predicates cannot be bound to fields that are not floating points (throws `ValidationException`) so accessing the NaN count should not happen if the field is missing from the stats struct.
On the other hand, these values may just be missing for a specific data file because they weren't collected. In this case, we may want to signal that the value is missing by returning the underlying null. This is why I left the `avg_value_size_in_bytes` column as an `Integer`.
If there is a case where we know we always have a value, then it makes sense to use the primitive. But it may also be better to have a method to check whether the value is known and still use a primitive when we may not have a value. I'm undecided at the moment and I'm waiting for actual uses. I'm also currently writing the evaluator and, while it isn't done, I think we may want to add more methods and keep the types more restrictive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598694349
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3604038869
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,313 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ // this end ID corresponds to stats base ID 200_000_000, the end of the stats range
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
+
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
+ }
+
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
+ }
+
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+ Map<Integer, Integer> parentIndex = TypeUtil.indexParents(tableSchema.asStruct());
+
+ for (int id : metricsConfig.metricsFieldIds()) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
+ Preconditions.checkArgument(
+ field != null, "Cannot build content stats schema: missing field ID %s", id);
Review Comment:
maybe mention that the field ID is missing from the table schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3581702772
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
+ return (FieldStats<T>) idToFieldStats.get(id);
+ }
+
+ public <T> void setStats(int id, FieldStats<T> fieldStats) {
+ Preconditions.checkArgument(
+ struct.field(StatsUtil.toBaseId(id)) != null,
+ "Cannot set stats for unknown field ID: %s",
+ id);
+ Preconditions.checkArgument(
+ id == fieldStats.fieldId(),
+ "Mismatched field stats for ID %s: actual ID %s",
+ id,
+ fieldStats.fieldId());
+
+ idToFieldStats.put(id, fieldStats);
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ return javaClass.cast(idToFieldStats.get(posToId[pos]));
Review Comment:
Yes, that behavior deviated from what `StructLike` instances are intended to do. If the indexes don't match then the reader is using a different schema than the class and we want that to fail. Part of the problem with the old implementation was not implementing the `StructLike` interface or using the `SupportsIndexProjection` base class correctly.
This implementation should bring the struct classes in line with what `SupportsIndexProjection` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589929482
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
+ return (FieldStats<T>) idToFieldStats.get(id);
+ }
+
+ public <T> void setStats(int id, FieldStats<T> fieldStats) {
+ Preconditions.checkArgument(
+ struct.field(StatsUtil.toBaseId(id)) != null,
+ "Cannot set stats for unknown field ID: %s",
+ id);
+ Preconditions.checkArgument(
+ id == fieldStats.fieldId(),
Review Comment:
`fieldId` will probably be removed. I don't like translating back into ID space and I don't think it is necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3590751274
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -46,17 +44,20 @@ interface FieldStats<T> extends StructLike {
boolean tightBounds();
/** The total value count, including null and NaN */
- Long valueCount();
+ long valueCount();
/** The total null value count */
- Long nullValueCount();
+ long nullValueCount();
/** The total NaN value count */
- Long nanValueCount();
+ long nanValueCount();
Review Comment:
consolidate the discussion on how to handle no stats scenario. there are two options
1. nullable boxed type of `Long`
2. primitive `long` + `ValidationException` + `hasXxx`
The existing APIs consistently use option 1, where `null` means "unknown/not collected," and callers null-check directly. If we decided to go with option 2, we should include the guardrails of `ValidationException` and `hasXxx`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3582805652
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -18,13 +18,13 @@
*/
package org.apache.iceberg;
-import java.util.List;
+import java.util.Set;
import org.apache.iceberg.types.Types;
-interface ContentStats extends StructLike {
+interface ContentStats {
/** A list of all the {@link FieldStats} */
- List<FieldStats<?>> fieldStats();
+ Iterable<FieldStats<?>> fieldStats();
Review Comment:
nit: the Javadoc still says `list`
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
+ return (FieldStats<T>) idToFieldStats.get(id);
+ }
+
+ public <T> void setStats(int id, FieldStats<T> fieldStats) {
+ Preconditions.checkArgument(
+ struct.field(StatsUtil.toBaseId(id)) != null,
+ "Cannot set stats for unknown field ID: %s",
+ id);
+ Preconditions.checkArgument(
+ id == fieldStats.fieldId(),
Review Comment:
since `fieldStats.fieldId()` is already there, maybe we don't need the first arg of `int id` for this method?
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -35,6 +35,12 @@ interface ContentStats extends StructLike {
*/
<T> FieldStats<T> statsFor(int fieldId);
- /** The stats struct holding nested structs with their respective field stats */
- Types.StructType statsStruct();
+ /** Container struct type containing tracked field-level stats structs. */
+ Types.StructType type();
+
+ /** Returns a copy, deep-copying all field stats. */
+ ContentStats copy();
+
+ /** Returns a copy, of only the selected field stats by ID. */
+ ContentStats copy(Set<Integer> fieldIds);
Review Comment:
is `project` a more accurate method name than `copy` here?
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
Review Comment:
why do we set to `null` for negative source value? won't it cause NPE in line 152 below?
```
@Override
public long nullValueCount() {
return nullValueCount;
}
```
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
Review Comment:
nit: `999_950` is derived — `(CONTENT_STATS_RANGE_END - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS = (200_000_000 - 10_000) / 200`. A short comment noting the derivation (or expressing it as that expression directly) would save readers from working it out.
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
+ switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> setLowerBound(value);
+ case StatsUtil.UPPER_BOUND_OFFSET -> setUpperBound(value);
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> this.tightBounds = (Boolean) value;
+ case StatsUtil.VALUE_COUNT_OFFSET -> this.valueCount = (Long) value;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> this.nullValueCount = (Long) value;
Review Comment:
the `FieldStats` interface changed the getter to return primitive type. not sure what's the reason there. With that, we need to check and prevent null object value here?
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
Review Comment:
I thought the setter wraps `byte[]` to `ByteBuffer`. should we check ByteBuffer here?
```
private void setLowerBound(Object lowerBound) {
this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
}
```
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
+ private static final Schema SCHEMA =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.StringType.get()),
+ optional(3, "score", Types.DoubleType.get()));
+
+ private static final Types.StructType CONTENT_STATS_STRUCT =
+ StatsUtil.statsReadSchema(SCHEMA, ImmutableList.of(1, 2, 3));
+ private static final Types.StructType UNKNOWN_FIELD_STATS_STRUCT =
+ StatsUtil.fieldStatsStruct(false, Types.IntegerType.get(), 10_800, MetricsModes.Full.get());
+
+ private static final FieldStats<Long> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("id").type().asStructType(), 0L, 25L, true, 26L, 0L, 0L, null);
+ private static final FieldStats<String> DATA_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("data").type().asStructType(),
+ "a",
+ "z",
+ true,
+ 26L,
+ 0L,
+ 0L,
+ null);
+
+ @Test
+ public void testEmptyContentStats() {
+ ContentStats stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.statsFor(1)).isNull();
+ assertThat(stats.statsFor(2)).isNull();
+ assertThat(stats.statsFor(3)).isNull();
+ assertThat(stats.statsFor(4)).as("Should ignore unknown field IDs").isNull();
+ }
+
+ @Test
+ public void testSetStats() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.statsFor(1)).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(2)).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(3)).isNull();
+ }
+
+ @Test
+ public void testSetStatsWrongId() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThatThrownBy(() -> stats.setStats(2, ID_STATS))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Mismatched field stats for ID 2: actual ID 1");
+ }
+
+ @Test
+ public void testSetStatsUnknownField() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ FieldStats<Integer> fieldStats =
+ new FieldStatsStruct<>(UNKNOWN_FIELD_STATS_STRUCT, 0, 10, false, 8, 3, 0, null);
+
+ assertThatThrownBy(() -> stats.setStats(4, fieldStats))
+ .hasMessage("Cannot set stats for unknown field ID: 4")
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
+ public void testGetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType contentStatsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(contentStatsStruct);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.get(0, FieldStats.class)).isEqualTo(DATA_STATS);
+ assertThat(stats.get(1, FieldStats.class)).isNull();
+ assertThat(stats.get(2, FieldStats.class)).isEqualTo(ID_STATS);
+ }
+
+ @Test
+ public void testSetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType statsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(statsStruct);
+
+ stats.set(1, null);
+ stats.set(2, ID_STATS);
+ stats.set(0, DATA_STATS);
+
+ assertThat(stats.statsFor(SCHEMA.findField("id").fieldId())).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("data").fieldId())).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("score").fieldId())).isNull();
+ }
+
+ @Test
+ public void testSize() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.size()).isEqualTo(3);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ ContentStats copy = stats.copy();
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(stats.type());
+
+ // each field stats is deep-copied using its own copy() method
+ assertThat(copy.statsFor(1)).isSameAs(idStatsCopy);
+ assertThat(copy.statsFor(2)).isSameAs(dataStatsCopy);
+ assertThat(copy.statsFor(3)).isNull();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testFilteredCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ // copy just the stats for field ID 2
+ ContentStats copy = stats.copy(ImmutableSet.of(2));
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(TypeUtil.select(stats.type(), ImmutableSet.of(1)));
+
Review Comment:
This assertion seems wrong.
`stats.type()`'s field IDs are stats base IDs (`fieldId * 200 + 10_000`, so 10200 / 10400 / 10600 for this schema), not table field IDs — so both `Set.of(1)` here and `Set.of(2)` on line 206 filter to an empty struct via `TypeUtil.select`. Empty equals empty, so the check passes regardless of what shape `copy.type()` actually has.
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
Review Comment:
nit: use `NUM_RESERVED_FIELD_STATS_IDS` instead of the literal `200`
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
}
- private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
- int fieldId =
- (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN
- + FIRST_SUPPORTED_METADATA_FIELD_ID;
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId) ? fieldId : -1;
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static Types.NestedField contentStatsFor(Schema schema) {
- ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor(schema);
- Types.NestedField result = TypeUtil.visit(schema, visitor);
- if (!visitor.skippedFieldIds.isEmpty()) {
- LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+
+ for (int id : metricsConfig.metricsFieldIds()) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
+
+ int baseId = toBaseId(id);
+ Types.StructType fieldStruct =
+ fieldStatsStruct(field.isOptional(), field.type(), baseId, metricsConfig.columnMode(id));
+
+ if (fieldStruct != null) {
+ fieldStructs.add(optional(baseId, fieldName, fieldStruct));
+ }
}
- return result;
+ return Types.StructType.of(fieldStructs);
}
- private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
- private final Schema tableSchema;
- private final List<Types.NestedField> statsFields = Lists.newArrayList();
- private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+ /**
+ * Produce a schema to read content stats for the given table field IDs.
+ *
+ * @param tableSchema a schema
+ * @param fieldIds an iterable of field IDs to project stats for
+ * @return a content stats struct for a read
+ */
+ public static Types.StructType statsReadSchema(Schema tableSchema, Iterable<Integer> fieldIds) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
- ContentStatsSchemaVisitor(Schema tableSchema) {
- this.tableSchema = tableSchema;
- }
+ for (int id : fieldIds) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
Review Comment:
`findField(id)` returns null for unknown IDs, so `field.isOptional()` on the next line NPEs. `statsReadSchema` is public with caller-supplied IDs . Skip-on-null or `Preconditions.checkArgument` with the ID in the message would be safer.
Line 137 in `statsWriteSchema` has the same problem but sources IDs from `metricsConfig`, so less exposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589939894
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
Review Comment:
Yeah, sorry. This was a mistake. We know that the value from `toCopy` is a byte buffer so we should not check the type. This should only copy the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3591835704
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
+ private static final Schema SCHEMA =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.StringType.get()),
+ optional(3, "score", Types.DoubleType.get()));
+
+ private static final Types.StructType CONTENT_STATS_STRUCT =
+ StatsUtil.statsReadSchema(SCHEMA, ImmutableList.of(1, 2, 3));
+ private static final Types.StructType UNKNOWN_FIELD_STATS_STRUCT =
+ StatsUtil.fieldStatsStruct(false, Types.IntegerType.get(), 10_800, MetricsModes.Full.get());
+
+ private static final FieldStats<Long> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("id").type().asStructType(), 0L, 25L, true, 26L, 0L, 0L, null);
+ private static final FieldStats<String> DATA_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("data").type().asStructType(),
+ "a",
+ "z",
+ true,
+ 26L,
+ 0L,
+ 0L,
+ null);
+
+ @Test
+ public void testEmptyContentStats() {
+ ContentStats stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.statsFor(1)).isNull();
+ assertThat(stats.statsFor(2)).isNull();
+ assertThat(stats.statsFor(3)).isNull();
+ assertThat(stats.statsFor(4)).as("Should ignore unknown field IDs").isNull();
+ }
+
+ @Test
+ public void testSetStats() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.statsFor(1)).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(2)).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(3)).isNull();
+ }
+
+ @Test
+ public void testSetStatsWrongId() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThatThrownBy(() -> stats.setStats(2, ID_STATS))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Mismatched field stats for ID 2: actual ID 1");
+ }
+
+ @Test
+ public void testSetStatsUnknownField() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ FieldStats<Integer> fieldStats =
+ new FieldStatsStruct<>(UNKNOWN_FIELD_STATS_STRUCT, 0, 10, false, 8, 3, 0, null);
+
+ assertThatThrownBy(() -> stats.setStats(4, fieldStats))
+ .hasMessage("Cannot set stats for unknown field ID: 4")
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
+ public void testGetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType contentStatsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(contentStatsStruct);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.get(0, FieldStats.class)).isEqualTo(DATA_STATS);
+ assertThat(stats.get(1, FieldStats.class)).isNull();
+ assertThat(stats.get(2, FieldStats.class)).isEqualTo(ID_STATS);
+ }
+
+ @Test
+ public void testSetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType statsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(statsStruct);
+
+ stats.set(1, null);
+ stats.set(2, ID_STATS);
+ stats.set(0, DATA_STATS);
+
+ assertThat(stats.statsFor(SCHEMA.findField("id").fieldId())).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("data").fieldId())).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("score").fieldId())).isNull();
+ }
+
+ @Test
+ public void testSize() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.size()).isEqualTo(3);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ ContentStats copy = stats.copy();
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(stats.type());
+
+ // each field stats is deep-copied using its own copy() method
+ assertThat(copy.statsFor(1)).isSameAs(idStatsCopy);
+ assertThat(copy.statsFor(2)).isSameAs(dataStatsCopy);
+ assertThat(copy.statsFor(3)).isNull();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testFilteredCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ // copy just the stats for field ID 2
+ ContentStats copy = stats.copy(ImmutableSet.of(2));
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(TypeUtil.select(stats.type(), ImmutableSet.of(1)));
+
Review Comment:
Yes, I think there is a bug in the code, so the test is passing vacuously. I left a [comment](https://github.com/apache/iceberg/pull/17159#discussion_r3591801368) in ContentsStatsStruct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589939894
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
Review Comment:
Yeah, sorry. This was a mistake. We know that the value from `toCopy` is a byte array so we should not check the type. This should only copy the value.
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3599082677
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
Review Comment:
Fixed.
I agree that we should skip a field if it is not present, just like unknown IDs are skipped in read schema generation.
Callers may derive field IDs from filters that contain metadata fields or other ID references. That is fine because filters can still run without stats. The result will be unknown and no filtering will occur. To be consistent, unknown IDs should be skipped here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3604336638
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, statsIds(fieldIds)) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ FieldStats<?> fieldToCopy = toCopy.idToFieldStats.get(fieldId);
+ if (fieldToCopy != null) {
+ idToFieldStats.put(fieldId, fieldToCopy.copy());
+ }
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
+ return (FieldStats<T>) idToFieldStats.get(id);
+ }
+
+ public <T> void setStats(int id, FieldStats<T> fieldStats) {
Review Comment:
```suggestion
public <T> void setStats(int fieldId, FieldStats<T> fieldStats) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589949471
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
Review Comment:
`null` means unknown internally, so it is correct.
Then the question is how we expose that unknown state to the caller. There's a thread with Amogh about this on the interface. The short answer is that I think we should introduce a method like `hasNullValueCount` to handle this rather than making `nullValueCount` return null values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3582922143
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
+ private static final Schema SCHEMA =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.StringType.get()),
+ optional(3, "score", Types.DoubleType.get()));
+
+ private static final Types.StructType CONTENT_STATS_STRUCT =
+ StatsUtil.statsReadSchema(SCHEMA, ImmutableList.of(1, 2, 3));
+ private static final Types.StructType UNKNOWN_FIELD_STATS_STRUCT =
+ StatsUtil.fieldStatsStruct(false, Types.IntegerType.get(), 10_800, MetricsModes.Full.get());
+
+ private static final FieldStats<Long> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("id").type().asStructType(), 0L, 25L, true, 26L, 0L, 0L, null);
+ private static final FieldStats<String> DATA_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("data").type().asStructType(),
+ "a",
+ "z",
+ true,
+ 26L,
+ 0L,
+ 0L,
+ null);
+
+ @Test
+ public void testEmptyContentStats() {
+ ContentStats stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.statsFor(1)).isNull();
+ assertThat(stats.statsFor(2)).isNull();
+ assertThat(stats.statsFor(3)).isNull();
+ assertThat(stats.statsFor(4)).as("Should ignore unknown field IDs").isNull();
+ }
+
+ @Test
+ public void testSetStats() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.statsFor(1)).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(2)).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(3)).isNull();
+ }
+
+ @Test
+ public void testSetStatsWrongId() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThatThrownBy(() -> stats.setStats(2, ID_STATS))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Mismatched field stats for ID 2: actual ID 1");
+ }
+
+ @Test
+ public void testSetStatsUnknownField() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ FieldStats<Integer> fieldStats =
+ new FieldStatsStruct<>(UNKNOWN_FIELD_STATS_STRUCT, 0, 10, false, 8, 3, 0, null);
+
+ assertThatThrownBy(() -> stats.setStats(4, fieldStats))
+ .hasMessage("Cannot set stats for unknown field ID: 4")
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
+ public void testGetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType contentStatsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(contentStatsStruct);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.get(0, FieldStats.class)).isEqualTo(DATA_STATS);
+ assertThat(stats.get(1, FieldStats.class)).isNull();
+ assertThat(stats.get(2, FieldStats.class)).isEqualTo(ID_STATS);
+ }
+
+ @Test
+ public void testSetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType statsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(statsStruct);
+
+ stats.set(1, null);
+ stats.set(2, ID_STATS);
+ stats.set(0, DATA_STATS);
+
+ assertThat(stats.statsFor(SCHEMA.findField("id").fieldId())).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("data").fieldId())).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("score").fieldId())).isNull();
+ }
+
+ @Test
+ public void testSize() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.size()).isEqualTo(3);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ ContentStats copy = stats.copy();
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(stats.type());
+
+ // each field stats is deep-copied using its own copy() method
+ assertThat(copy.statsFor(1)).isSameAs(idStatsCopy);
+ assertThat(copy.statsFor(2)).isSameAs(dataStatsCopy);
+ assertThat(copy.statsFor(3)).isNull();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testFilteredCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ // copy just the stats for field ID 2
+ ContentStats copy = stats.copy(ImmutableSet.of(2));
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(TypeUtil.select(stats.type(), ImmutableSet.of(1)));
+
Review Comment:
This assertion on L209 seems wrong.
`stats.type()`'s field IDs are stats base IDs (`fieldId * 200 + 10_000`, so 10200 / 10400 / 10600 for this schema), not table field IDs — so both `Set.of(1)` here and `Set.of(2)` on line 206 filter to an empty struct via `TypeUtil.select`. Empty equals empty, so the check passes regardless of what shape `copy.type()` actually has.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3603603842
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
Review Comment:
we used to check that upper/lower bound are of the right type and would throw `Invalid lower bound type, expected a subtype of xx`. Are we not doing that anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3599058719
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
Review Comment:
It was. There was a bad test that Steven found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562170937
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -35,6 +34,9 @@ interface ContentStats extends StructLike {
*/
<T> FieldStats<T> statsFor(int fieldId);
- /** The stats struct holding nested structs with their respective field stats */
- Types.StructType statsStruct();
+ /** Container struct type containing tracked field-level stats structs. */
+ Types.StructType type();
+
+ /** Returns a copy of these {@link ContentStats}, deep-copying the contained field stats. */
+ ContentStats copy();
Review Comment:
Needed to implement copy in `TrackedFile` correctly. We also need to follow up with methods to project by field ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562194855
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -113,7 +140,7 @@ void testDataFileAdapterDelegation() {
assertThat(dataFile.manifestLocation()).isEqualTo(MANIFEST_LOCATION);
assertThat(dataFile.equalityFieldIds()).isNull();
assertThat(dataFile.columnSizes()).isNull();
- assertThat(dataFile.valueCounts()).containsOnly(Map.entry(1, 100L), Map.entry(2, 200L));
+ assertThat(dataFile.valueCounts()).containsOnly(Map.entry(1, 100L), Map.entry(2, 100L));
Review Comment:
Updated to align with what stats would actually look like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3580235586
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
Review Comment:
Based on the comment below the change to make this API a `long` was a conscious one but given the additional change to MetricsUtil that's made wouldn't this NPE in cases where value / null / nan counts are null?
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
+ }
+ } else {
+ for (Map.Entry<Integer, FieldStats<?>> entry : toCopy.idToFieldStats.entrySet()) {
+ idToFieldStats.put(entry.getKey(), entry.getValue().copy());
+ }
+ }
+ }
+
+ @Override
+ public Iterable<FieldStats<?>> fieldStats() {
+ return idToFieldStats.values();
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> FieldStats<T> statsFor(int id) {
+ return (FieldStats<T>) idToFieldStats.get(id);
+ }
+
+ public <T> void setStats(int id, FieldStats<T> fieldStats) {
+ Preconditions.checkArgument(
+ struct.field(StatsUtil.toBaseId(id)) != null,
+ "Cannot set stats for unknown field ID: %s",
+ id);
+ Preconditions.checkArgument(
+ id == fieldStats.fieldId(),
+ "Mismatched field stats for ID %s: actual ID %s",
+ id,
+ fieldStats.fieldId());
+
+ idToFieldStats.put(id, fieldStats);
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ return javaClass.cast(idToFieldStats.get(posToId[pos]));
Review Comment:
`BaseContentStats` get used to check `pos` against the struct size, and it looks like the old code returned null in case the pos was "out of bounds" of the struct. It looks like that was done for Avro because on the write path Avro calls get() for every field in the writer schema. I don't think we're fully plumbed through which is why we don't see any issues but are we handling that case differently in this PR, or we're just deferring that? If I recall correctly we wanted to use this in memory representation regardless of format version (because we theoretically could adapt the existing stats structure to this)
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -35,6 +35,12 @@ interface ContentStats extends StructLike {
*/
<T> FieldStats<T> statsFor(int fieldId);
- /** The stats struct holding nested structs with their respective field stats */
- Types.StructType statsStruct();
+ /** Container struct type containing tracked field-level stats structs. */
+ Types.StructType type();
+
+ /** Returns a copy, deep-copying all field stats. */
Review Comment:
nit: Returns a deep copy of all field stats?
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -46,17 +44,20 @@ interface FieldStats<T> extends StructLike {
boolean tightBounds();
/** The total value count, including null and NaN */
- Long valueCount();
+ long valueCount();
/** The total null value count */
- Long nullValueCount();
+ long nullValueCount();
/** The total NaN value count */
- Long nanValueCount();
+ long nanValueCount();
Review Comment:
Curious what's driving this, from a spec perspective is there an open question about if these stats should be required? My understanding of the parquet spec is that value counts are required but null/NaN counts are optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598797763
##########
core/src/test/java/org/apache/iceberg/TestStatsUtil.java:
##########
@@ -18,511 +18,449 @@
*/
package org.apache.iceberg;
-import static org.apache.iceberg.FieldStatistic.AVG_VALUE_SIZE_IN_BYTES;
-import static org.apache.iceberg.FieldStatistic.LOWER_BOUND;
-import static org.apache.iceberg.FieldStatistic.NAN_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.NULL_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.TIGHT_BOUNDS;
-import static org.apache.iceberg.FieldStatistic.UPPER_BOUND;
-import static org.apache.iceberg.FieldStatistic.VALUE_COUNT;
-import static org.apache.iceberg.types.Types.NestedField.optional;
-import static org.apache.iceberg.types.Types.NestedField.required;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.stream.IntStream;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
public class TestStatsUtil {
- // the reserved field IDs from the reserved field ID space as defined in
- // https://iceberg.apache.org/spec/#reserved-field-ids
- private static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - 200;
+ @Test
+ public void testToBaseIdWithData() {
+ assertThat(StatsUtil.toBaseId(0)).isEqualTo(10_000);
+ assertThat(StatsUtil.toBaseId(1)).isEqualTo(10_200);
+ assertThat(StatsUtil.toBaseId(100)).isEqualTo(30_000);
+ assertThat(StatsUtil.toBaseId(999_949)).isEqualTo(199_999_800);
+ }
@Test
- public void statsIdsForTableColumns() {
- int offset = 0;
- for (int id = 0; id < StatsUtil.MAX_DATA_FIELD_ID; id++) {
- int statsFieldId = StatsUtil.statsFieldIdForField(id);
- int expected = StatsUtil.STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + offset;
- assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
- offset += StatsUtil.NUM_SUPPORTED_STATS_PER_COLUMN;
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).as("at pos %s", id).isEqualTo(id);
- }
+ public void testToBaseIdWithMetadata() {
+ assertThat(StatsUtil.toBaseId(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId()))
+ .isEqualTo(9_000);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.ROW_ID.fieldId())).isEqualTo(9_200);
+ }
- // also verify hardcoded field IDs from docs
- int fieldId = 0;
- int statsFieldId = 10_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 1;
- statsFieldId = 10_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 2;
- statsFieldId = 10_400;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 5;
- statsFieldId = 11_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 100;
- statsFieldId = 30_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = StatsUtil.MAX_DATA_FIELD_ID;
- statsFieldId = StatsUtil.MAX_DATA_STATS_FIELD_ID;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = -1;
- statsFieldId = -1;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
+ @Test
+ public void testBaseIdUnassigned() {
+ assertThat(StatsUtil.toBaseId(999_950)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(-1)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.FILE_PATH.fieldId())).isEqualTo(-1);
}
@Test
- public void statsIdsOverflowForTableColumns() {
- // pick 100 random IDs that are > MAX_FIELD_ID and < RESERVED_FIELD_IDS_START as going over
- // the entire ID range takes too long
- int invalidFieldId = -1;
- for (int i = 0; i < 100; i++) {
- int id =
- ThreadLocalRandom.current()
- .nextInt(StatsUtil.MAX_DATA_FIELD_ID + 1, RESERVED_FIELD_IDS_START);
- assertThat(StatsUtil.statsFieldIdForField(id)).as("at pos %s", id).isEqualTo(invalidFieldId);
- }
+ public void testToFieldIdWithData() {
+ assertThat(StatsUtil.toFieldId(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.toFieldId(10_200)).isEqualTo(1);
+ assertThat(StatsUtil.toFieldId(30_000)).isEqualTo(100);
+ assertThat(StatsUtil.toFieldId(199_999_800)).isEqualTo(999_949);
+ }
- assertThat(StatsUtil.fieldIdForStatsField(-1)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(0)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(200)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(5_000)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(8_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_001)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_201)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_500)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_900)).isEqualTo(invalidFieldId);
-
- // stats field IDs at or above the exclusive upper bound are invalid
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END + 200))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(Integer.MAX_VALUE)).isEqualTo(invalidFieldId);
-
- // field ID just past MAX_DATA_FIELD_ID is invalid
- assertThat(StatsUtil.statsFieldIdForField(StatsUtil.MAX_DATA_FIELD_ID + 1))
- .isEqualTo(invalidFieldId);
+ @Test
+ public void testToFieldIdWithMetadata() {
+ assertThat(StatsUtil.toFieldId(9_000))
+ .isEqualTo(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId());
+ assertThat(StatsUtil.toFieldId(9_200)).isEqualTo(MetadataColumns.ROW_ID.fieldId());
}
@Test
- public void statsIdsForMetadataColumns() {
- int fieldId = MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
- int statsFieldId = 9_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = MetadataColumns.ROW_ID.fieldId();
- statsFieldId = 9_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- // reserved metadata fields with IDs below/above the reserved stats range have no stats
- int invalidFieldId = -1;
- assertThat(StatsUtil.fieldIdForStatsField(8_800)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_400)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_800)).isEqualTo(invalidFieldId);
- assertThat(IntStream.range(RESERVED_FIELD_IDS_START, Integer.MAX_VALUE))
- .filteredOn(id -> !StatsUtil.SUPPORTED_METADATA_FIELD_IDS.contains(id))
- .allSatisfy(
- id ->
- assertThat(StatsUtil.statsFieldIdForField(id))
- .as("at pos %s", id)
- .isEqualTo(invalidFieldId));
+ public void testToFieldIdOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
}
@Test
- public void contentStatsForSimpleSchema() {
- Types.NestedField intField = required(0, "i", Types.IntegerType.get());
- Types.NestedField floatField = required(2, "f", Types.FloatType.get());
- Types.NestedField stringField = required(4, "s", Types.StringType.get());
- Types.NestedField booleanField = required(6, "b", Types.BooleanType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema = new Schema(intField, floatField, stringField, booleanField, uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(intField, 10000)),
- optional(10400, "f", FieldStatistic.fieldStatsFor(floatField, 10400)),
- optional(10800, "s", FieldStatistic.fieldStatsFor(stringField, 10800)),
- optional(11200, "b", FieldStatistic.fieldStatsFor(booleanField, 11200)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testToFieldIdReservedMetadataRange() {
+ // 9,000 to 10,000 (exclusive) is reserved for metadata column stats
+ assertThatThrownBy(() -> StatsUtil.toFieldId(9_400))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Unsupported metadata stats field ID: 9400");
}
@Test
- public void contentStatsForComplexSchema() {
- Types.NestedField listElement = optional(3, "element", Types.IntegerType.get());
- Types.NestedField structInt = optional(7, "int", Types.IntegerType.get());
- Types.NestedField structString = optional(8, "string", Types.StringType.get());
- Types.NestedField mapKey = required(22, "key", Types.IntegerType.get());
- Types.NestedField mapValue = optional(24, "value", Types.StringType.get());
- Types.NestedField variantField = required(30, "variant", Types.VariantType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema =
- new Schema(
- required(0, "i", Types.IntegerType.get()),
- required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
- required(
- 6,
- "simple_struct",
- Types.StructType.of(
- optional(7, "int", Types.IntegerType.get()),
- optional(8, "string", Types.StringType.get()))),
- required(
- 20,
- "b",
- Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
- variantField,
- uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(
- 10000,
- "i",
- FieldStatistic.fieldStatsFor(
- required(0, "i", Types.IntegerType.get()), 10000)),
- optional(
- 10600, "list.element", FieldStatistic.fieldStatsFor(listElement, 10600)),
- optional(
- 11400, "simple_struct.int", FieldStatistic.fieldStatsFor(structInt, 11400)),
- optional(
- 11600,
- "simple_struct.string",
- FieldStatistic.fieldStatsFor(structString, 11600)),
- optional(14400, "b.key", FieldStatistic.fieldStatsFor(mapKey, 14400)),
- optional(14800, "b.value", FieldStatistic.fieldStatsFor(mapValue, 14800)),
- optional(16000, "variant", FieldStatistic.fieldStatsFor(variantField, 16000)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testStatOffset() {
+ assertThat(StatsUtil.statOffset(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(10_201)).isEqualTo(1);
+ assertThat(StatsUtil.statOffset(10_211)).isEqualTo(11);
+ assertThat(StatsUtil.statOffset(10_399)).isEqualTo(199);
+ assertThat(StatsUtil.statOffset(10_400)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(199_999_999)).isEqualTo(199);
}
@Test
- public void contentStatsChildNamesAreUnique() {
- Schema schema =
- new Schema(
- required(1, "a", Types.StructType.of(required(2, "x", Types.IntegerType.get()))),
- required(3, "b", Types.StructType.of(required(4, "x", Types.IntegerType.get()))));
+ public void testStatOffsetOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
+ }
- Types.StructType contentStats = StatsUtil.contentStatsFor(schema).type().asStructType();
+ private static final List<Type> FIXED_WIDTH_TYPES =
+ List.of(
+ Types.BooleanType.get(),
+ Types.IntegerType.get(),
+ Types.LongType.get(),
+ Types.DateType.get(),
+ Types.TimeType.get(),
+ Types.TimestampType.withoutZone(),
+ Types.TimestampType.withZone(),
+ Types.TimestampNanoType.withoutZone(),
+ Types.TimestampNanoType.withZone(),
+ Types.DecimalType.of(9, 2),
+ Types.UUIDType.get(),
+ Types.FixedType.ofLength(16));
+
+ @ParameterizedTest
+ @FieldSource("FIXED_WIDTH_TYPES")
+ public void testFixedWidthPrimitiveStruct(Type type) {
+ // fixed-width, non-floating-point types track bounds (including tight bounds) but have no NaN
+ // count or average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(contentStats.fields().stream().map(Types.NestedField::name).toList())
- .doesNotHaveDuplicates()
- .containsExactly("a.x", "b.x");
+ private static final List<Type> VARIABLE_WIDTH_TYPES =
+ List.of(Types.StringType.get(), Types.BinaryType.get());
+
+ @ParameterizedTest
+ @FieldSource("VARIABLE_WIDTH_TYPES")
+ public void testVariableWidthPrimitiveStruct(Type type) {
+ // variable-width types track bounds (including tight bounds) and an average value size, but no
+ // NaN count
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void contentStatsSkipsFieldsOutsideStatsRange() {
- Types.NestedField validField = required(0, "i", Types.IntegerType.get());
- Types.NestedField outOfRangeField =
- required(StatsUtil.MAX_DATA_FIELD_ID + 1, "out_of_range", Types.IntegerType.get());
- Schema schema = new Schema(validField, outOfRangeField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(validField, 10000)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ private static final List<Type> GEO_TYPES =
+ List.of(
+ Types.GeometryType.crs84(),
+ Types.GeometryType.of("srid:3857"),
+ Types.GeographyType.crs84(),
+ Types.GeographyType.of("srid:4269"));
+
+ @ParameterizedTest
+ @FieldSource("GEO_TYPES")
+ public void testGeoStruct(Type type) {
+ // geometry and geography use bounding-box structs for their bounds, do not track tight bounds,
+ // and record an average value size
+ Types.StructType lowerBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_010, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_011, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_012, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_013, "m", Types.DoubleType.get()));
+ Types.StructType upperBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_014, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_015, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_016, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_017, "m", Types.DoubleType.get()));
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", lowerBound),
+ Types.NestedField.optional(30_002, "upper_bound", upperBound),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void conditionalFieldInclusionForInteger() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.IntegerType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName())
- .doesNotContain(
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ private static final List<Type> FLOATING_POINT_TYPES =
+ List.of(Types.FloatType.get(), Types.DoubleType.get());
+
+ @ParameterizedTest
+ @FieldSource("FLOATING_POINT_TYPES")
+ public void testFloatingPointStruct(Type type) {
+ // floating-point types track bounds (including tight bounds) and a NaN count, but have no
+ // average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_006, "nan_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.IntegerType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName())
- .doesNotContain(NAN_VALUE_COUNT.fieldName(), AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ private static final Types.StructType POINT =
+ Types.StructType.of(
+ Types.NestedField.required(1, "x", Types.FloatType.get()),
+ Types.NestedField.required(2, "y", Types.FloatType.get()));
+
+ private static final List<Type> NESTED_TYPES =
+ List.of(
+ Types.ListType.ofRequired(1, Types.IntegerType.get()),
+ Types.ListType.ofOptional(1, POINT),
+ Types.MapType.ofRequired(1, 2, Types.StringType.get(), Types.StringType.get()),
+ Types.MapType.ofOptional(1, 2, Types.StringType.get(), POINT));
+
+ @ParameterizedTest
+ @FieldSource("NESTED_TYPES")
+ public void testNestedTypesHaveNoStats(Type type) {
+ // list and map types are not tracked and produce no stats struct
+ assertThat(StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get())).isNull();
}
@Test
- public void conditionalFieldInclusionForFloatAndDouble() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.FloatType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), AVG_VALUE_SIZE_IN_BYTES.fieldName());
-
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.DoubleType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName());
+ public void testRequiredField() {
+ // a required column does not produce a null_value_count field
+ Type type = Types.IntegerType.get();
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(false, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
@Test
- public void conditionalFieldInclusionForString() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.StringType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), NAN_VALUE_COUNT.fieldName());
-
+ public void testStringNoneMode() {
+ // none mode produces no stats struct
assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.StringType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ StatsUtil.fieldStatsStruct(
+ true, Types.StringType.get(), 30_000, MetricsModes.None.get()))
+ .isNull();
}
@Test
- public void conditionalFieldInclusionForBinary() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.BinaryType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NAN_VALUE_COUNT.fieldName());
+ public void testStringCountsMode() {
+ // counts mode drops the bounds (and tight_bounds) but keeps counts and the value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, Types.StringType.get(), 30_000, MetricsModes.Counts.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.BinaryType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), NAN_VALUE_COUNT.fieldName());
+ private static final List<MetricsModes.MetricsMode> BOUNDS_MODES =
+ List.of(MetricsModes.Truncate.withLength(16), MetricsModes.Full.get());
+
+ @ParameterizedTest
+ @FieldSource("BOUNDS_MODES")
+ public void testStringBoundsModes(MetricsModes.MetricsMode mode) {
+ // truncate and full both retain the full bounds struct
+ Type string = Types.StringType.get();
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", string),
+ Types.NestedField.optional(30_002, "upper_bound", string),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual = StatsUtil.fieldStatsStruct(true, string, 30_000, mode);
+
+ assertSameStructure(expected, actual);
}
@Test
- public void conditionalFieldInclusionForGeometry() {
- Types.StructType requiredStats =
- FieldStatistic.fieldStatsFor(required(1, "x", Types.GeometryType.crs84()), 10000);
- assertThat(fieldStatsNames(requiredStats))
- .containsExactly(LOWER_BOUND.fieldName(), UPPER_BOUND.fieldName(), VALUE_COUNT.fieldName())
- .doesNotContain(
- TIGHT_BOUNDS.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
- assertGeoBoundStructs(requiredStats, 10000);
-
- Types.StructType optionalStats =
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.GeometryType.crs84()), 10000);
- assertThat(fieldStatsNames(optionalStats))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName())
- .doesNotContain(
- TIGHT_BOUNDS.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
- assertGeoBoundStructs(optionalStats, 10000);
+ public void testContentStatsReadSchema() {
Review Comment:
I'll add a test, but the majority of many of these cases aren't relevant because the field types are tested directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598713082
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
}
- private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
- int fieldId =
- (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN
- + FIRST_SUPPORTED_METADATA_FIELD_ID;
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId) ? fieldId : -1;
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static Types.NestedField contentStatsFor(Schema schema) {
- ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor(schema);
- Types.NestedField result = TypeUtil.visit(schema, visitor);
- if (!visitor.skippedFieldIds.isEmpty()) {
- LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+
+ for (int id : metricsConfig.metricsFieldIds()) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
+
+ int baseId = toBaseId(id);
+ Types.StructType fieldStruct =
+ fieldStatsStruct(field.isOptional(), field.type(), baseId, metricsConfig.columnMode(id));
+
+ if (fieldStruct != null) {
+ fieldStructs.add(optional(baseId, fieldName, fieldStruct));
+ }
}
- return result;
+ return Types.StructType.of(fieldStructs);
}
- private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
- private final Schema tableSchema;
- private final List<Types.NestedField> statsFields = Lists.newArrayList();
- private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+ /**
+ * Produce a schema to read content stats for the given table field IDs.
+ *
+ * @param tableSchema a schema
+ * @param fieldIds an iterable of field IDs to project stats for
+ * @return a content stats struct for a read
+ */
+ public static Types.StructType statsReadSchema(Schema tableSchema, Iterable<Integer> fieldIds) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
- ContentStatsSchemaVisitor(Schema tableSchema) {
- this.tableSchema = tableSchema;
- }
+ for (int id : fieldIds) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
Review Comment:
Added a check in both places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3599057093
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileStruct.java:
##########
@@ -211,6 +217,103 @@ void copy() {
assertThat(copy.recordCount()).isEqualTo(50L);
assertThat(copy.fileSizeInBytes()).isEqualTo(512L);
assertThat(copy.specId()).isEqualTo(1);
+ assertThat(copy.contentStats()).isSameAs(CONTENT_STATS_COPY);
+ assertThat(copy.sortOrderId()).isEqualTo(5);
+ assertThat(copy.deletionVector()).isSameAs(DELETION_VECTOR_COPY);
+ assertThat(copy.manifestInfo()).isSameAs(MANIFEST_INFO_COPY);
+ assertThat(copy.keyMetadata()).isEqualTo(ByteBuffer.wrap(new byte[] {1, 2, 3}));
+ assertThat(copy.splitOffsets()).containsExactly(100L, 200L);
+ assertThat(copy.equalityIds()).containsExactly(1, 2, 3);
+ assertThat(copy.partition()).isSameAs(PARTITION_COPY);
+
+ // mutable fields are deep-copied, not shared with the original
+ assertThat(copy.keyMetadata()).isNotSameAs(file.keyMetadata());
+ }
+
+ @Test
+ void copyWithStats() {
+ ContentStats stats = Mockito.mock(ContentStats.class);
+ ContentStats statsCopy = Mockito.mock(ContentStats.class);
+ Mockito.when(stats.copy(ImmutableSet.of(1))).thenReturn(statsCopy);
+
+ TrackedFileStruct file =
+ new TrackedFileStruct(
+ TRACKING,
+ FileContent.DATA,
+ FORMAT_VERSION_V4,
+ "s3://bucket/data/00000-0-file.parquet",
+ FileFormat.PARQUET,
+ PARTITION,
+ 50L,
+ 512L,
+ 1,
+ stats,
+ 5,
+ DELETION_VECTOR,
+ MANIFEST_INFO,
+ ByteBuffer.wrap(new byte[] {1, 2, 3}),
+ ImmutableList.of(100L, 200L),
+ ImmutableList.of(1, 2, 3));
+
+ TrackedFile copy = file.copyWithStats(ImmutableSet.of(1));
+
+ assertThat(copy).isInstanceOf(TrackedFileStruct.class);
+ assertThat(copy.tracking()).isSameAs(TRACKING_COPY);
+ assertThat(copy.contentType()).isEqualTo(FileContent.DATA);
+ assertThat(copy.formatVersion()).isEqualTo(FORMAT_VERSION_V4);
+ assertThat(copy.location()).isEqualTo("s3://bucket/data/00000-0-file.parquet");
+ assertThat(copy.fileFormat()).isEqualTo(FileFormat.PARQUET);
+ assertThat(copy.recordCount()).isEqualTo(50L);
+ assertThat(copy.fileSizeInBytes()).isEqualTo(512L);
+ assertThat(copy.specId()).isEqualTo(1);
+ assertThat(copy.contentStats()).isSameAs(statsCopy);
+ assertThat(copy.sortOrderId()).isEqualTo(5);
+ assertThat(copy.deletionVector()).isSameAs(DELETION_VECTOR_COPY);
+ assertThat(copy.manifestInfo()).isSameAs(MANIFEST_INFO_COPY);
+ assertThat(copy.keyMetadata()).isEqualTo(ByteBuffer.wrap(new byte[] {1, 2, 3}));
+ assertThat(copy.splitOffsets()).containsExactly(100L, 200L);
+ assertThat(copy.equalityIds()).containsExactly(1, 2, 3);
+ assertThat(copy.partition()).isSameAs(PARTITION_COPY);
+
+ // mutable fields are deep-copied, not shared with the original
+ assertThat(copy.keyMetadata()).isNotSameAs(file.keyMetadata());
+ }
+
+ @Test
+ void copyWithoutStats() {
Review Comment:
Good catch.
Fixed. I thought that `withStats` was being set but it turns out the interface has a default that passes `ImmutableSet.of()` instead. I removed the unnecessary boolean and updated this test to verify that no methods are called on `stats`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3603940497
##########
core/src/test/java/org/apache/iceberg/TestStatsUtil.java:
##########
@@ -18,511 +18,578 @@
*/
package org.apache.iceberg;
-import static org.apache.iceberg.FieldStatistic.AVG_VALUE_SIZE_IN_BYTES;
-import static org.apache.iceberg.FieldStatistic.LOWER_BOUND;
-import static org.apache.iceberg.FieldStatistic.NAN_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.NULL_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.TIGHT_BOUNDS;
-import static org.apache.iceberg.FieldStatistic.UPPER_BOUND;
-import static org.apache.iceberg.FieldStatistic.VALUE_COUNT;
-import static org.apache.iceberg.types.Types.NestedField.optional;
-import static org.apache.iceberg.types.Types.NestedField.required;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.stream.IntStream;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
public class TestStatsUtil {
- // the reserved field IDs from the reserved field ID space as defined in
- // https://iceberg.apache.org/spec/#reserved-field-ids
- private static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - 200;
+ @Test
+ public void testToBaseIdWithData() {
+ assertThat(StatsUtil.toBaseId(0)).isEqualTo(10_000);
+ assertThat(StatsUtil.toBaseId(1)).isEqualTo(10_200);
+ assertThat(StatsUtil.toBaseId(100)).isEqualTo(30_000);
+ assertThat(StatsUtil.toBaseId(999_949)).isEqualTo(199_999_800);
+ }
@Test
- public void statsIdsForTableColumns() {
- int offset = 0;
- for (int id = 0; id < StatsUtil.MAX_DATA_FIELD_ID; id++) {
- int statsFieldId = StatsUtil.statsFieldIdForField(id);
- int expected = StatsUtil.STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + offset;
- assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
- offset += StatsUtil.NUM_SUPPORTED_STATS_PER_COLUMN;
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).as("at pos %s", id).isEqualTo(id);
- }
+ public void testToBaseIdWithMetadata() {
+ assertThat(StatsUtil.toBaseId(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId()))
+ .isEqualTo(9_000);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.ROW_ID.fieldId())).isEqualTo(9_200);
+ }
- // also verify hardcoded field IDs from docs
- int fieldId = 0;
- int statsFieldId = 10_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 1;
- statsFieldId = 10_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 2;
- statsFieldId = 10_400;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 5;
- statsFieldId = 11_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 100;
- statsFieldId = 30_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = StatsUtil.MAX_DATA_FIELD_ID;
- statsFieldId = StatsUtil.MAX_DATA_STATS_FIELD_ID;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = -1;
- statsFieldId = -1;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
+ @Test
+ public void testBaseIdUnassigned() {
+ assertThat(StatsUtil.toBaseId(999_950)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(-1)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.FILE_PATH.fieldId())).isEqualTo(-1);
}
@Test
- public void statsIdsOverflowForTableColumns() {
- // pick 100 random IDs that are > MAX_FIELD_ID and < RESERVED_FIELD_IDS_START as going over
- // the entire ID range takes too long
- int invalidFieldId = -1;
- for (int i = 0; i < 100; i++) {
- int id =
- ThreadLocalRandom.current()
- .nextInt(StatsUtil.MAX_DATA_FIELD_ID + 1, RESERVED_FIELD_IDS_START);
- assertThat(StatsUtil.statsFieldIdForField(id)).as("at pos %s", id).isEqualTo(invalidFieldId);
- }
+ public void testToFieldIdWithData() {
+ assertThat(StatsUtil.toFieldId(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.toFieldId(10_200)).isEqualTo(1);
+ assertThat(StatsUtil.toFieldId(30_000)).isEqualTo(100);
+ assertThat(StatsUtil.toFieldId(199_999_800)).isEqualTo(999_949);
+ }
- assertThat(StatsUtil.fieldIdForStatsField(-1)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(0)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(200)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(5_000)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(8_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_001)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_201)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_500)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_900)).isEqualTo(invalidFieldId);
-
- // stats field IDs at or above the exclusive upper bound are invalid
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END + 200))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(Integer.MAX_VALUE)).isEqualTo(invalidFieldId);
-
- // field ID just past MAX_DATA_FIELD_ID is invalid
- assertThat(StatsUtil.statsFieldIdForField(StatsUtil.MAX_DATA_FIELD_ID + 1))
- .isEqualTo(invalidFieldId);
+ @Test
+ public void testToFieldIdWithMetadata() {
+ assertThat(StatsUtil.toFieldId(9_000))
+ .isEqualTo(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId());
+ assertThat(StatsUtil.toFieldId(9_200)).isEqualTo(MetadataColumns.ROW_ID.fieldId());
}
@Test
- public void statsIdsForMetadataColumns() {
- int fieldId = MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
- int statsFieldId = 9_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = MetadataColumns.ROW_ID.fieldId();
- statsFieldId = 9_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- // reserved metadata fields with IDs below/above the reserved stats range have no stats
- int invalidFieldId = -1;
- assertThat(StatsUtil.fieldIdForStatsField(8_800)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_400)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_800)).isEqualTo(invalidFieldId);
- assertThat(IntStream.range(RESERVED_FIELD_IDS_START, Integer.MAX_VALUE))
- .filteredOn(id -> !StatsUtil.SUPPORTED_METADATA_FIELD_IDS.contains(id))
- .allSatisfy(
- id ->
- assertThat(StatsUtil.statsFieldIdForField(id))
- .as("at pos %s", id)
- .isEqualTo(invalidFieldId));
+ public void testToFieldIdOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
}
@Test
- public void contentStatsForSimpleSchema() {
- Types.NestedField intField = required(0, "i", Types.IntegerType.get());
- Types.NestedField floatField = required(2, "f", Types.FloatType.get());
- Types.NestedField stringField = required(4, "s", Types.StringType.get());
- Types.NestedField booleanField = required(6, "b", Types.BooleanType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema = new Schema(intField, floatField, stringField, booleanField, uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(intField, 10000)),
- optional(10400, "f", FieldStatistic.fieldStatsFor(floatField, 10400)),
- optional(10800, "s", FieldStatistic.fieldStatsFor(stringField, 10800)),
- optional(11200, "b", FieldStatistic.fieldStatsFor(booleanField, 11200)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testToFieldIdReservedMetadataRange() {
+ // 9,000 to 10,000 (exclusive) is reserved for metadata column stats
+ assertThatThrownBy(() -> StatsUtil.toFieldId(9_400))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Unsupported metadata stats field ID: 9400");
}
@Test
- public void contentStatsForComplexSchema() {
- Types.NestedField listElement = optional(3, "element", Types.IntegerType.get());
- Types.NestedField structInt = optional(7, "int", Types.IntegerType.get());
- Types.NestedField structString = optional(8, "string", Types.StringType.get());
- Types.NestedField mapKey = required(22, "key", Types.IntegerType.get());
- Types.NestedField mapValue = optional(24, "value", Types.StringType.get());
- Types.NestedField variantField = required(30, "variant", Types.VariantType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema =
- new Schema(
- required(0, "i", Types.IntegerType.get()),
- required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
- required(
- 6,
- "simple_struct",
- Types.StructType.of(
- optional(7, "int", Types.IntegerType.get()),
- optional(8, "string", Types.StringType.get()))),
- required(
- 20,
- "b",
- Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
- variantField,
- uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(
- 10000,
- "i",
- FieldStatistic.fieldStatsFor(
- required(0, "i", Types.IntegerType.get()), 10000)),
- optional(
- 10600, "list.element", FieldStatistic.fieldStatsFor(listElement, 10600)),
- optional(
- 11400, "simple_struct.int", FieldStatistic.fieldStatsFor(structInt, 11400)),
- optional(
- 11600,
- "simple_struct.string",
- FieldStatistic.fieldStatsFor(structString, 11600)),
- optional(14400, "b.key", FieldStatistic.fieldStatsFor(mapKey, 14400)),
- optional(14800, "b.value", FieldStatistic.fieldStatsFor(mapValue, 14800)),
- optional(16000, "variant", FieldStatistic.fieldStatsFor(variantField, 16000)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testStatOffset() {
+ assertThat(StatsUtil.statOffset(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(10_201)).isEqualTo(1);
+ assertThat(StatsUtil.statOffset(10_211)).isEqualTo(11);
+ assertThat(StatsUtil.statOffset(10_399)).isEqualTo(199);
+ assertThat(StatsUtil.statOffset(10_400)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(199_999_999)).isEqualTo(199);
}
@Test
- public void contentStatsChildNamesAreUnique() {
- Schema schema =
- new Schema(
- required(1, "a", Types.StructType.of(required(2, "x", Types.IntegerType.get()))),
- required(3, "b", Types.StructType.of(required(4, "x", Types.IntegerType.get()))));
+ public void testStatOffsetOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
+ }
- Types.StructType contentStats = StatsUtil.contentStatsFor(schema).type().asStructType();
+ private static final List<Type> FIXED_WIDTH_TYPES =
+ List.of(
+ Types.BooleanType.get(),
+ Types.IntegerType.get(),
+ Types.LongType.get(),
+ Types.DateType.get(),
+ Types.TimeType.get(),
+ Types.TimestampType.withoutZone(),
+ Types.TimestampType.withZone(),
+ Types.TimestampNanoType.withoutZone(),
+ Types.TimestampNanoType.withZone(),
+ Types.DecimalType.of(9, 2),
+ Types.UUIDType.get(),
+ Types.FixedType.ofLength(16));
+
+ @ParameterizedTest
+ @FieldSource("FIXED_WIDTH_TYPES")
+ public void testFixedWidthPrimitiveStruct(Type type) {
+ // fixed-width, non-floating-point types track bounds (including tight bounds) but have no NaN
+ // count or average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(contentStats.fields().stream().map(Types.NestedField::name).toList())
- .doesNotHaveDuplicates()
- .containsExactly("a.x", "b.x");
+ private static final List<Type> VARIABLE_WIDTH_TYPES =
+ List.of(Types.StringType.get(), Types.BinaryType.get());
+
+ @ParameterizedTest
+ @FieldSource("VARIABLE_WIDTH_TYPES")
+ public void testVariableWidthPrimitiveStruct(Type type) {
+ // variable-width types track bounds (including tight bounds) and an average value size, but no
+ // NaN count
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void contentStatsSkipsFieldsOutsideStatsRange() {
- Types.NestedField validField = required(0, "i", Types.IntegerType.get());
- Types.NestedField outOfRangeField =
- required(StatsUtil.MAX_DATA_FIELD_ID + 1, "out_of_range", Types.IntegerType.get());
- Schema schema = new Schema(validField, outOfRangeField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(validField, 10000)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ private static final List<Type> GEO_TYPES =
+ List.of(
+ Types.GeometryType.crs84(),
+ Types.GeometryType.of("srid:3857"),
+ Types.GeographyType.crs84(),
+ Types.GeographyType.of("srid:4269"));
+
+ @ParameterizedTest
+ @FieldSource("GEO_TYPES")
+ public void testGeoStruct(Type type) {
+ // geometry and geography use bounding-box structs for their bounds, do not track tight bounds,
+ // and record an average value size
+ Types.StructType lowerBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_010, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_011, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_012, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_013, "m", Types.DoubleType.get()));
+ Types.StructType upperBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_014, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_015, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_016, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_017, "m", Types.DoubleType.get()));
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", lowerBound),
+ Types.NestedField.optional(30_002, "upper_bound", upperBound),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
Review Comment:
for Geo the spec says that `avg_value_size_in_bytes` should be omitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3605435776
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,313 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ // this end ID corresponds to stats base ID 200_000_000, the end of the stats range
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
+
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
+ }
+
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
+ }
+
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+ Map<Integer, Integer> parentIndex = TypeUtil.indexParents(tableSchema.asStruct());
+
+ for (int id : metricsConfig.metricsFieldIds()) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
+ Preconditions.checkArgument(
+ field != null, "Cannot build content stats schema: missing field ID %s", id);
Review Comment:
I think that's what this says.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3605431184
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
Review Comment:
That's correct. We don't want additional type checks. The readers and writers are responsible for ensuring types match. Those are generated from the schema and won't operate on invalid values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3605504963
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.lowerBound)
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.upperBound)
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
+ switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> setLowerBound(value);
+ case StatsUtil.UPPER_BOUND_OFFSET -> setUpperBound(value);
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> this.tightBounds = (Boolean) value;
Review Comment:
I think the only time that tight bounds will be null is when the column is not present. Otherwise, we should always have a value for tight bounds because we can always assume that bounds are not tight if it is unknown.
That said, the column is currently `optional`. I think we should decide whether to make that required or to accept a `null` here and translate it to `false` since the bounds are assumed to not be tight when this is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3591103535
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -516,8 +516,11 @@ static Map<Integer, Long> nanValueCounts(ContentStats stats) {
Map<Integer, Long> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
- if (fs != null && fs.nanValueCount() != null) {
- result.put(fs.fieldId(), fs.nanValueCount());
+ if (fs != null) {
+ Type boundType = fs.type().fieldType("lower_bound");
+ if (boundType.typeId() == Type.TypeID.FLOAT || boundType.typeId() == Type.TypeID.DOUBLE) {
Review Comment:
**Problem:**
`boundType.typeId()` NPEs when the stats struct has no `lower_bound` field. This happens for float/double columns in Counts mode — `StatsUtil.fieldStatsStruct` only adds `lower_bound` when `mode.hasBounds()` is true (L262-273), but `nan_value_count` is added for any floating-point field regardless of mode.
because bound fields are conditionally omitted from the stats struct, the field's *type* isn't available to consumers when bounds aren't collected. `MetricsUtil` has no way to know whether a column is float/double if `lower_bound` is missing.
**Options:**
1. **Tactical fix — use a different presence signal.** `fs.type().field("nan_value_count") != null` maps directly to "this field can have NaN counts" and doesn't depend on `lower_bound` being present. Local, targeted.
2. **Keep bound fields in the schema even when values aren't collected.** Change `fieldStatsStruct` so Counts mode still emits `lower_bound` / `upper_bound` NestedFields (with null values). Preserves the type/values distinction, keeps schema shape stable across modes, and consumers can always derive the bound type. Null column probably has negligible cost for Parquet manifest.
Option 2 is the architecturally cleanest. Option 1 is the smallest diff.
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -531,8 +534,11 @@ static Map<Integer, ByteBuffer> lowerBounds(ContentStats stats) {
Map<Integer, ByteBuffer> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
- if (fs != null && fs.lowerBound() != null && fs.type() != null) {
- result.put(fs.fieldId(), Conversions.toByteBuffer(fs.type(), fs.lowerBound()));
+ if (fs != null) {
+ Type boundType = fs.type().fieldType("lower_bound");
+ if (fs.lowerBound() != null && boundType != null) {
+ result.put(fs.fieldId(), Conversions.toByteBuffer(boundType, fs.lowerBound()));
Review Comment:
**Summary:** `FieldStats.type()` changed from "field's data type" to "stats struct type," which affects only geo — for a geo column, `lower_bound`'s schema type is a struct (bounding-box) rather than a scalar matching the field type, and `Conversions.toByteBuffer` has no `case STRUCT`, so it throws instead of dispatching to `case GEOMETRY`.
**Layout.** For a geo column, `lower_bound` / `upper_bound` are bounding-box structs, not scalar Geometry values (`geoLowerBound` / `geoUpperBound` at StatsUtil.java:207-237):
```
FieldStatsStruct for a GEOMETRY column:
├── lower_bound: STRUCT ← name is "lower_bound", type is a struct
│ ├── x: double (required)
│ ├── y: double (required)
│ ├── z: double (optional)
│ └── m: double (optional)
├── upper_bound: STRUCT ← same shape
├── tight_bounds: boolean
├── value_count: long
└── null_value_count: long
```
**FieldStats.type() before/after.** Before this PR: returned the field's data type (`GeometryType` for geo). After (FieldStats.java:32): returns the stats struct type; the bound's type is now retrieved via `fs.type().fieldType("lower_bound")`.
**Runtime failure.** For geo, `fs.type().fieldType("lower_bound")` returns the geo_lower `StructType`. `Conversions.toByteBuffer` (Conversions.java:95-146) switches on `typeId()` — `STRUCT` has no matching case, so `default` throws `UnsupportedOperationException: Cannot serialize type: STRUCT`. The `case GEOMETRY` / `case GEOGRAPHY` branch handling bounds via `((GeospatialBound) value).toByteBuffer()` is unreachable.
Same at line 558 for `upperBounds`.
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileStruct.java:
##########
@@ -211,6 +217,103 @@ void copy() {
assertThat(copy.recordCount()).isEqualTo(50L);
assertThat(copy.fileSizeInBytes()).isEqualTo(512L);
assertThat(copy.specId()).isEqualTo(1);
+ assertThat(copy.contentStats()).isSameAs(CONTENT_STATS_COPY);
+ assertThat(copy.sortOrderId()).isEqualTo(5);
+ assertThat(copy.deletionVector()).isSameAs(DELETION_VECTOR_COPY);
+ assertThat(copy.manifestInfo()).isSameAs(MANIFEST_INFO_COPY);
+ assertThat(copy.keyMetadata()).isEqualTo(ByteBuffer.wrap(new byte[] {1, 2, 3}));
+ assertThat(copy.splitOffsets()).containsExactly(100L, 200L);
+ assertThat(copy.equalityIds()).containsExactly(1, 2, 3);
+ assertThat(copy.partition()).isSameAs(PARTITION_COPY);
+
+ // mutable fields are deep-copied, not shared with the original
+ assertThat(copy.keyMetadata()).isNotSameAs(file.keyMetadata());
+ }
+
+ @Test
+ void copyWithStats() {
+ ContentStats stats = Mockito.mock(ContentStats.class);
+ ContentStats statsCopy = Mockito.mock(ContentStats.class);
+ Mockito.when(stats.copy(ImmutableSet.of(1))).thenReturn(statsCopy);
+
+ TrackedFileStruct file =
+ new TrackedFileStruct(
+ TRACKING,
+ FileContent.DATA,
+ FORMAT_VERSION_V4,
+ "s3://bucket/data/00000-0-file.parquet",
+ FileFormat.PARQUET,
+ PARTITION,
+ 50L,
+ 512L,
+ 1,
+ stats,
+ 5,
+ DELETION_VECTOR,
+ MANIFEST_INFO,
+ ByteBuffer.wrap(new byte[] {1, 2, 3}),
+ ImmutableList.of(100L, 200L),
+ ImmutableList.of(1, 2, 3));
+
+ TrackedFile copy = file.copyWithStats(ImmutableSet.of(1));
+
+ assertThat(copy).isInstanceOf(TrackedFileStruct.class);
+ assertThat(copy.tracking()).isSameAs(TRACKING_COPY);
+ assertThat(copy.contentType()).isEqualTo(FileContent.DATA);
+ assertThat(copy.formatVersion()).isEqualTo(FORMAT_VERSION_V4);
+ assertThat(copy.location()).isEqualTo("s3://bucket/data/00000-0-file.parquet");
+ assertThat(copy.fileFormat()).isEqualTo(FileFormat.PARQUET);
+ assertThat(copy.recordCount()).isEqualTo(50L);
+ assertThat(copy.fileSizeInBytes()).isEqualTo(512L);
+ assertThat(copy.specId()).isEqualTo(1);
+ assertThat(copy.contentStats()).isSameAs(statsCopy);
+ assertThat(copy.sortOrderId()).isEqualTo(5);
+ assertThat(copy.deletionVector()).isSameAs(DELETION_VECTOR_COPY);
+ assertThat(copy.manifestInfo()).isSameAs(MANIFEST_INFO_COPY);
+ assertThat(copy.keyMetadata()).isEqualTo(ByteBuffer.wrap(new byte[] {1, 2, 3}));
+ assertThat(copy.splitOffsets()).containsExactly(100L, 200L);
+ assertThat(copy.equalityIds()).containsExactly(1, 2, 3);
+ assertThat(copy.partition()).isSameAs(PARTITION_COPY);
+
+ // mutable fields are deep-copied, not shared with the original
+ assertThat(copy.keyMetadata()).isNotSameAs(file.keyMetadata());
+ }
+
+ @Test
+ void copyWithoutStats() {
Review Comment:
Two problems:
1. **The test at line 283 doesn't call `copyWithoutStats()`.** Line 305 calls `copyWithStats(ImmutableSet.of(1))`. The `copy.contentStats() == null` assertion on 316 only passes because `stats` is a Mockito mock and `.copy(Set)` was never stubbed — not a real check of `copyWithoutStats` semantics.
2. **`TrackedFile.copyWithoutStats()` throws at runtime.** The default at TrackedFile.java:180 calls `copyWithStats(Collections.emptySet())` → `ContentStatsStruct.copy(emptySet)` → trips the `!fieldIds.isEmpty()` precondition at ContentStatsStruct.java:105 → `IllegalArgumentException`. Any file with non-null `contentStats` throws.
Verified with [this gist](https://gist.github.com/stevenzwu/97a9de101e4500b7b7398702d10eb923).
Two fix options: loosen the `ContentStatsStruct.copy` precondition to accept empty sets, or short-circuit `copyWithoutStats` in `TrackedFileStruct` to null out `contentStats` directly. Later seems more reasonable.
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -531,8 +534,11 @@ static Map<Integer, ByteBuffer> lowerBounds(ContentStats stats) {
Map<Integer, ByteBuffer> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
- if (fs != null && fs.lowerBound() != null && fs.type() != null) {
- result.put(fs.fieldId(), Conversions.toByteBuffer(fs.type(), fs.lowerBound()));
+ if (fs != null) {
+ Type boundType = fs.type().fieldType("lower_bound");
+ if (fs.lowerBound() != null && boundType != null) {
+ result.put(fs.fieldId(), Conversions.toByteBuffer(boundType, fs.lowerBound()));
Review Comment:
**Coverage gap.** `TestFieldStatsStruct.testGeoSerialization` exercises Java / Kryo / InternalData round-trip but not this MetricsUtil path. `TestTrackedFileAdapters` calls `MetricsUtil.lowerBounds` via `TrackedFileAdapters.asDataFile`, but its schema is `(id: int, score: float)` — no geo. Geo × MetricsUtil isn't tested anywhere.
Should we add `TestMetricsUtil`? Every other v4 stats class in this PR has its own test file (`TestFieldStatsStruct`, `TestContentStatsStruct`, `TestStatsUtil`) — MetricsUtil is the exception and currently has no direct test. A new `core/src/test/java/org/apache/iceberg/TestMetricsUtil.java` would be the natural home for:
- `testLowerBoundsGeo` — geometry column, assert `ByteBuffer` output matches `GeospatialBound.toByteBuffer()`
- `testUpperBoundsGeography` — same for geography
- `testNanValueCountsWithCountsMode` — float column in Counts mode, assert no NPE (see the sibling thread on L521)
- `testValueCountsProjection` — `FieldStatsStruct` constructed via the projection constructor, verify unset counts are handled without NPE (see the sibling thread on L490)
- `testLowerBoundsScalarPrimitives` — smoke test across int / long / double / string / binary / decimal / uuid
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
Review Comment:
I asked AI to compare the coverage btw the deleted `TestContentStats` and this new class `TestContentStatsStruct`. This flagged one is probably worth a callout.
**Coverage regression: bound-type validation at set time is gone.**
Deleted `TestContentStats.setByPositionWithInvalidLowerAndUpperBound` asserted that setting a wrong-typed bound threw `IllegalArgumentException` with `"Invalid lower bound type, expected a subtype of ..."`. Neither `FieldStatsStruct.setLowerBound` (FieldStatsStruct.java:108-110) nor `ContentStatsStruct.setStats` (ContentStatsStruct.java:64-76) performs this check now. Compile-time generics catch typed callers; raw-type / `StructLike.set` / reflection paths bypass.
Test that would lock in the invariant if validation were restored (verified locally: fails on the current PR):
```java
@Test
void testSetLowerBoundRejectsWrongType() {
Types.StructType idStatsSchema =
StatsUtil.statsReadSchema(
new Schema(required(1, "id", Types.IntegerType.get())),
ImmutableList.of(1))
.field("id").type().asStructType();
FieldStatsStruct<Object> stats = new FieldStatsStruct<>(idStatsSchema);
assertThatThrownBy(() -> stats.set(0, 5.0)) // lower_bound offset, Double vs Integer
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("expected a subtype of java.lang.Integer: java.lang.Double");
}
```
Without the check, wrong-typed bounds flow through silently and fail later at `Conversions.toByteBuffer` with a less-actionable `ClassCastException`.
Intentional (readers guarantee types upstream), or a gap to restore at `setLowerBound` / `setUpperBound`?
##########
core/src/test/java/org/apache/iceberg/TestStatsUtil.java:
##########
@@ -18,511 +18,449 @@
*/
package org.apache.iceberg;
-import static org.apache.iceberg.FieldStatistic.AVG_VALUE_SIZE_IN_BYTES;
-import static org.apache.iceberg.FieldStatistic.LOWER_BOUND;
-import static org.apache.iceberg.FieldStatistic.NAN_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.NULL_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.TIGHT_BOUNDS;
-import static org.apache.iceberg.FieldStatistic.UPPER_BOUND;
-import static org.apache.iceberg.FieldStatistic.VALUE_COUNT;
-import static org.apache.iceberg.types.Types.NestedField.optional;
-import static org.apache.iceberg.types.Types.NestedField.required;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.stream.IntStream;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
public class TestStatsUtil {
- // the reserved field IDs from the reserved field ID space as defined in
- // https://iceberg.apache.org/spec/#reserved-field-ids
- private static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - 200;
+ @Test
+ public void testToBaseIdWithData() {
+ assertThat(StatsUtil.toBaseId(0)).isEqualTo(10_000);
+ assertThat(StatsUtil.toBaseId(1)).isEqualTo(10_200);
+ assertThat(StatsUtil.toBaseId(100)).isEqualTo(30_000);
+ assertThat(StatsUtil.toBaseId(999_949)).isEqualTo(199_999_800);
+ }
@Test
- public void statsIdsForTableColumns() {
- int offset = 0;
- for (int id = 0; id < StatsUtil.MAX_DATA_FIELD_ID; id++) {
- int statsFieldId = StatsUtil.statsFieldIdForField(id);
- int expected = StatsUtil.STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + offset;
- assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
- offset += StatsUtil.NUM_SUPPORTED_STATS_PER_COLUMN;
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).as("at pos %s", id).isEqualTo(id);
- }
+ public void testToBaseIdWithMetadata() {
+ assertThat(StatsUtil.toBaseId(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId()))
+ .isEqualTo(9_000);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.ROW_ID.fieldId())).isEqualTo(9_200);
+ }
- // also verify hardcoded field IDs from docs
- int fieldId = 0;
- int statsFieldId = 10_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 1;
- statsFieldId = 10_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 2;
- statsFieldId = 10_400;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 5;
- statsFieldId = 11_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 100;
- statsFieldId = 30_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = StatsUtil.MAX_DATA_FIELD_ID;
- statsFieldId = StatsUtil.MAX_DATA_STATS_FIELD_ID;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = -1;
- statsFieldId = -1;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
+ @Test
+ public void testBaseIdUnassigned() {
+ assertThat(StatsUtil.toBaseId(999_950)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(-1)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.FILE_PATH.fieldId())).isEqualTo(-1);
}
@Test
- public void statsIdsOverflowForTableColumns() {
- // pick 100 random IDs that are > MAX_FIELD_ID and < RESERVED_FIELD_IDS_START as going over
- // the entire ID range takes too long
- int invalidFieldId = -1;
- for (int i = 0; i < 100; i++) {
- int id =
- ThreadLocalRandom.current()
- .nextInt(StatsUtil.MAX_DATA_FIELD_ID + 1, RESERVED_FIELD_IDS_START);
- assertThat(StatsUtil.statsFieldIdForField(id)).as("at pos %s", id).isEqualTo(invalidFieldId);
- }
+ public void testToFieldIdWithData() {
+ assertThat(StatsUtil.toFieldId(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.toFieldId(10_200)).isEqualTo(1);
+ assertThat(StatsUtil.toFieldId(30_000)).isEqualTo(100);
+ assertThat(StatsUtil.toFieldId(199_999_800)).isEqualTo(999_949);
+ }
- assertThat(StatsUtil.fieldIdForStatsField(-1)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(0)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(200)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(5_000)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(8_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_001)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_201)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_500)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_900)).isEqualTo(invalidFieldId);
-
- // stats field IDs at or above the exclusive upper bound are invalid
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END + 200))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(Integer.MAX_VALUE)).isEqualTo(invalidFieldId);
-
- // field ID just past MAX_DATA_FIELD_ID is invalid
- assertThat(StatsUtil.statsFieldIdForField(StatsUtil.MAX_DATA_FIELD_ID + 1))
- .isEqualTo(invalidFieldId);
+ @Test
+ public void testToFieldIdWithMetadata() {
+ assertThat(StatsUtil.toFieldId(9_000))
+ .isEqualTo(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId());
+ assertThat(StatsUtil.toFieldId(9_200)).isEqualTo(MetadataColumns.ROW_ID.fieldId());
}
@Test
- public void statsIdsForMetadataColumns() {
- int fieldId = MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
- int statsFieldId = 9_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = MetadataColumns.ROW_ID.fieldId();
- statsFieldId = 9_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- // reserved metadata fields with IDs below/above the reserved stats range have no stats
- int invalidFieldId = -1;
- assertThat(StatsUtil.fieldIdForStatsField(8_800)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_400)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_800)).isEqualTo(invalidFieldId);
- assertThat(IntStream.range(RESERVED_FIELD_IDS_START, Integer.MAX_VALUE))
- .filteredOn(id -> !StatsUtil.SUPPORTED_METADATA_FIELD_IDS.contains(id))
- .allSatisfy(
- id ->
- assertThat(StatsUtil.statsFieldIdForField(id))
- .as("at pos %s", id)
- .isEqualTo(invalidFieldId));
+ public void testToFieldIdOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
}
@Test
- public void contentStatsForSimpleSchema() {
- Types.NestedField intField = required(0, "i", Types.IntegerType.get());
- Types.NestedField floatField = required(2, "f", Types.FloatType.get());
- Types.NestedField stringField = required(4, "s", Types.StringType.get());
- Types.NestedField booleanField = required(6, "b", Types.BooleanType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema = new Schema(intField, floatField, stringField, booleanField, uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(intField, 10000)),
- optional(10400, "f", FieldStatistic.fieldStatsFor(floatField, 10400)),
- optional(10800, "s", FieldStatistic.fieldStatsFor(stringField, 10800)),
- optional(11200, "b", FieldStatistic.fieldStatsFor(booleanField, 11200)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testToFieldIdReservedMetadataRange() {
+ // 9,000 to 10,000 (exclusive) is reserved for metadata column stats
+ assertThatThrownBy(() -> StatsUtil.toFieldId(9_400))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Unsupported metadata stats field ID: 9400");
}
@Test
- public void contentStatsForComplexSchema() {
- Types.NestedField listElement = optional(3, "element", Types.IntegerType.get());
- Types.NestedField structInt = optional(7, "int", Types.IntegerType.get());
- Types.NestedField structString = optional(8, "string", Types.StringType.get());
- Types.NestedField mapKey = required(22, "key", Types.IntegerType.get());
- Types.NestedField mapValue = optional(24, "value", Types.StringType.get());
- Types.NestedField variantField = required(30, "variant", Types.VariantType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema =
- new Schema(
- required(0, "i", Types.IntegerType.get()),
- required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
- required(
- 6,
- "simple_struct",
- Types.StructType.of(
- optional(7, "int", Types.IntegerType.get()),
- optional(8, "string", Types.StringType.get()))),
- required(
- 20,
- "b",
- Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
- variantField,
- uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(
- 10000,
- "i",
- FieldStatistic.fieldStatsFor(
- required(0, "i", Types.IntegerType.get()), 10000)),
- optional(
- 10600, "list.element", FieldStatistic.fieldStatsFor(listElement, 10600)),
- optional(
- 11400, "simple_struct.int", FieldStatistic.fieldStatsFor(structInt, 11400)),
- optional(
- 11600,
- "simple_struct.string",
- FieldStatistic.fieldStatsFor(structString, 11600)),
- optional(14400, "b.key", FieldStatistic.fieldStatsFor(mapKey, 14400)),
- optional(14800, "b.value", FieldStatistic.fieldStatsFor(mapValue, 14800)),
- optional(16000, "variant", FieldStatistic.fieldStatsFor(variantField, 16000)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testStatOffset() {
+ assertThat(StatsUtil.statOffset(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(10_201)).isEqualTo(1);
+ assertThat(StatsUtil.statOffset(10_211)).isEqualTo(11);
+ assertThat(StatsUtil.statOffset(10_399)).isEqualTo(199);
+ assertThat(StatsUtil.statOffset(10_400)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(199_999_999)).isEqualTo(199);
}
@Test
- public void contentStatsChildNamesAreUnique() {
- Schema schema =
- new Schema(
- required(1, "a", Types.StructType.of(required(2, "x", Types.IntegerType.get()))),
- required(3, "b", Types.StructType.of(required(4, "x", Types.IntegerType.get()))));
+ public void testStatOffsetOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
+ }
- Types.StructType contentStats = StatsUtil.contentStatsFor(schema).type().asStructType();
+ private static final List<Type> FIXED_WIDTH_TYPES =
+ List.of(
+ Types.BooleanType.get(),
+ Types.IntegerType.get(),
+ Types.LongType.get(),
+ Types.DateType.get(),
+ Types.TimeType.get(),
+ Types.TimestampType.withoutZone(),
+ Types.TimestampType.withZone(),
+ Types.TimestampNanoType.withoutZone(),
+ Types.TimestampNanoType.withZone(),
+ Types.DecimalType.of(9, 2),
+ Types.UUIDType.get(),
+ Types.FixedType.ofLength(16));
+
+ @ParameterizedTest
+ @FieldSource("FIXED_WIDTH_TYPES")
+ public void testFixedWidthPrimitiveStruct(Type type) {
+ // fixed-width, non-floating-point types track bounds (including tight bounds) but have no NaN
+ // count or average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(contentStats.fields().stream().map(Types.NestedField::name).toList())
- .doesNotHaveDuplicates()
- .containsExactly("a.x", "b.x");
+ private static final List<Type> VARIABLE_WIDTH_TYPES =
+ List.of(Types.StringType.get(), Types.BinaryType.get());
+
+ @ParameterizedTest
+ @FieldSource("VARIABLE_WIDTH_TYPES")
+ public void testVariableWidthPrimitiveStruct(Type type) {
+ // variable-width types track bounds (including tight bounds) and an average value size, but no
+ // NaN count
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void contentStatsSkipsFieldsOutsideStatsRange() {
- Types.NestedField validField = required(0, "i", Types.IntegerType.get());
- Types.NestedField outOfRangeField =
- required(StatsUtil.MAX_DATA_FIELD_ID + 1, "out_of_range", Types.IntegerType.get());
- Schema schema = new Schema(validField, outOfRangeField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(validField, 10000)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ private static final List<Type> GEO_TYPES =
+ List.of(
+ Types.GeometryType.crs84(),
+ Types.GeometryType.of("srid:3857"),
+ Types.GeographyType.crs84(),
+ Types.GeographyType.of("srid:4269"));
+
+ @ParameterizedTest
+ @FieldSource("GEO_TYPES")
+ public void testGeoStruct(Type type) {
+ // geometry and geography use bounding-box structs for their bounds, do not track tight bounds,
+ // and record an average value size
+ Types.StructType lowerBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_010, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_011, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_012, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_013, "m", Types.DoubleType.get()));
+ Types.StructType upperBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_014, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_015, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_016, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_017, "m", Types.DoubleType.get()));
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", lowerBound),
+ Types.NestedField.optional(30_002, "upper_bound", upperBound),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void conditionalFieldInclusionForInteger() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.IntegerType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName())
- .doesNotContain(
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ private static final List<Type> FLOATING_POINT_TYPES =
+ List.of(Types.FloatType.get(), Types.DoubleType.get());
+
+ @ParameterizedTest
+ @FieldSource("FLOATING_POINT_TYPES")
+ public void testFloatingPointStruct(Type type) {
+ // floating-point types track bounds (including tight bounds) and a NaN count, but have no
+ // average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_006, "nan_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.IntegerType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName())
- .doesNotContain(NAN_VALUE_COUNT.fieldName(), AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ private static final Types.StructType POINT =
+ Types.StructType.of(
+ Types.NestedField.required(1, "x", Types.FloatType.get()),
+ Types.NestedField.required(2, "y", Types.FloatType.get()));
+
+ private static final List<Type> NESTED_TYPES =
+ List.of(
+ Types.ListType.ofRequired(1, Types.IntegerType.get()),
+ Types.ListType.ofOptional(1, POINT),
+ Types.MapType.ofRequired(1, 2, Types.StringType.get(), Types.StringType.get()),
+ Types.MapType.ofOptional(1, 2, Types.StringType.get(), POINT));
+
+ @ParameterizedTest
+ @FieldSource("NESTED_TYPES")
+ public void testNestedTypesHaveNoStats(Type type) {
+ // list and map types are not tracked and produce no stats struct
+ assertThat(StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get())).isNull();
}
@Test
- public void conditionalFieldInclusionForFloatAndDouble() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.FloatType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), AVG_VALUE_SIZE_IN_BYTES.fieldName());
-
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.DoubleType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName());
+ public void testRequiredField() {
+ // a required column does not produce a null_value_count field
+ Type type = Types.IntegerType.get();
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(false, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
@Test
- public void conditionalFieldInclusionForString() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.StringType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), NAN_VALUE_COUNT.fieldName());
-
+ public void testStringNoneMode() {
+ // none mode produces no stats struct
assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.StringType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
+ StatsUtil.fieldStatsStruct(
+ true, Types.StringType.get(), 30_000, MetricsModes.None.get()))
+ .isNull();
}
@Test
- public void conditionalFieldInclusionForBinary() {
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.BinaryType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NAN_VALUE_COUNT.fieldName());
+ public void testStringCountsMode() {
+ // counts mode drops the bounds (and tight_bounds) but keeps counts and the value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, Types.StringType.get(), 30_000, MetricsModes.Counts.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(
- fieldStatsNames(
- FieldStatistic.fieldStatsFor(required(1, "x", Types.BinaryType.get()), 10000)))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- TIGHT_BOUNDS.fieldName(),
- VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName())
- .doesNotContain(NULL_VALUE_COUNT.fieldName(), NAN_VALUE_COUNT.fieldName());
+ private static final List<MetricsModes.MetricsMode> BOUNDS_MODES =
+ List.of(MetricsModes.Truncate.withLength(16), MetricsModes.Full.get());
+
+ @ParameterizedTest
+ @FieldSource("BOUNDS_MODES")
+ public void testStringBoundsModes(MetricsModes.MetricsMode mode) {
+ // truncate and full both retain the full bounds struct
+ Type string = Types.StringType.get();
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", string),
+ Types.NestedField.optional(30_002, "upper_bound", string),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual = StatsUtil.fieldStatsStruct(true, string, 30_000, mode);
+
+ assertSameStructure(expected, actual);
}
@Test
- public void conditionalFieldInclusionForGeometry() {
- Types.StructType requiredStats =
- FieldStatistic.fieldStatsFor(required(1, "x", Types.GeometryType.crs84()), 10000);
- assertThat(fieldStatsNames(requiredStats))
- .containsExactly(LOWER_BOUND.fieldName(), UPPER_BOUND.fieldName(), VALUE_COUNT.fieldName())
- .doesNotContain(
- TIGHT_BOUNDS.fieldName(),
- NULL_VALUE_COUNT.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
- assertGeoBoundStructs(requiredStats, 10000);
-
- Types.StructType optionalStats =
- FieldStatistic.fieldStatsFor(optional(1, "x", Types.GeometryType.crs84()), 10000);
- assertThat(fieldStatsNames(optionalStats))
- .containsExactly(
- LOWER_BOUND.fieldName(),
- UPPER_BOUND.fieldName(),
- VALUE_COUNT.fieldName(),
- NULL_VALUE_COUNT.fieldName())
- .doesNotContain(
- TIGHT_BOUNDS.fieldName(),
- NAN_VALUE_COUNT.fieldName(),
- AVG_VALUE_SIZE_IN_BYTES.fieldName());
- assertGeoBoundStructs(optionalStats, 10000);
+ public void testContentStatsReadSchema() {
Review Comment:
**Coverage gap:** `statsWriteSchema` has no tests. `statsReadSchema` has four dedicated tests (this one + `testContentStatsReadSchemaSubset` / `NestedStruct` / `OmitsListAndMap`), but the new write-side entry point — which branches on per-column `metricsConfig.columnMode(id)` — is not exercised in a schema-generation test.
The branching behavior worth covering:
- Default mode = Full for all columns → parallel to `testContentStatsReadSchema`
- Per-column override (e.g., one column Counts, one Full) → the mode-per-column dispatch
- One column set to None → skipped in the output struct
- Sorted columns → default upgraded to Truncate (see `sortedColumnDefaultMode` at MetricsConfig.java:326)
Without these, the per-column `columnMode(id)` path (StatsUtil.java:141) has zero regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3605433752
##########
core/src/test/java/org/apache/iceberg/TestStatsUtil.java:
##########
@@ -18,511 +18,578 @@
*/
package org.apache.iceberg;
-import static org.apache.iceberg.FieldStatistic.AVG_VALUE_SIZE_IN_BYTES;
-import static org.apache.iceberg.FieldStatistic.LOWER_BOUND;
-import static org.apache.iceberg.FieldStatistic.NAN_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.NULL_VALUE_COUNT;
-import static org.apache.iceberg.FieldStatistic.TIGHT_BOUNDS;
-import static org.apache.iceberg.FieldStatistic.UPPER_BOUND;
-import static org.apache.iceberg.FieldStatistic.VALUE_COUNT;
-import static org.apache.iceberg.types.Types.NestedField.optional;
-import static org.apache.iceberg.types.Types.NestedField.required;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.stream.IntStream;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
public class TestStatsUtil {
- // the reserved field IDs from the reserved field ID space as defined in
- // https://iceberg.apache.org/spec/#reserved-field-ids
- private static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - 200;
+ @Test
+ public void testToBaseIdWithData() {
+ assertThat(StatsUtil.toBaseId(0)).isEqualTo(10_000);
+ assertThat(StatsUtil.toBaseId(1)).isEqualTo(10_200);
+ assertThat(StatsUtil.toBaseId(100)).isEqualTo(30_000);
+ assertThat(StatsUtil.toBaseId(999_949)).isEqualTo(199_999_800);
+ }
@Test
- public void statsIdsForTableColumns() {
- int offset = 0;
- for (int id = 0; id < StatsUtil.MAX_DATA_FIELD_ID; id++) {
- int statsFieldId = StatsUtil.statsFieldIdForField(id);
- int expected = StatsUtil.STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + offset;
- assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
- offset += StatsUtil.NUM_SUPPORTED_STATS_PER_COLUMN;
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).as("at pos %s", id).isEqualTo(id);
- }
+ public void testToBaseIdWithMetadata() {
+ assertThat(StatsUtil.toBaseId(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId()))
+ .isEqualTo(9_000);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.ROW_ID.fieldId())).isEqualTo(9_200);
+ }
- // also verify hardcoded field IDs from docs
- int fieldId = 0;
- int statsFieldId = 10_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 1;
- statsFieldId = 10_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 2;
- statsFieldId = 10_400;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 5;
- statsFieldId = 11_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = 100;
- statsFieldId = 30_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = StatsUtil.MAX_DATA_FIELD_ID;
- statsFieldId = StatsUtil.MAX_DATA_STATS_FIELD_ID;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = -1;
- statsFieldId = -1;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
+ @Test
+ public void testBaseIdUnassigned() {
+ assertThat(StatsUtil.toBaseId(999_950)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(-1)).isEqualTo(-1);
+ assertThat(StatsUtil.toBaseId(MetadataColumns.FILE_PATH.fieldId())).isEqualTo(-1);
}
@Test
- public void statsIdsOverflowForTableColumns() {
- // pick 100 random IDs that are > MAX_FIELD_ID and < RESERVED_FIELD_IDS_START as going over
- // the entire ID range takes too long
- int invalidFieldId = -1;
- for (int i = 0; i < 100; i++) {
- int id =
- ThreadLocalRandom.current()
- .nextInt(StatsUtil.MAX_DATA_FIELD_ID + 1, RESERVED_FIELD_IDS_START);
- assertThat(StatsUtil.statsFieldIdForField(id)).as("at pos %s", id).isEqualTo(invalidFieldId);
- }
+ public void testToFieldIdWithData() {
+ assertThat(StatsUtil.toFieldId(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.toFieldId(10_200)).isEqualTo(1);
+ assertThat(StatsUtil.toFieldId(30_000)).isEqualTo(100);
+ assertThat(StatsUtil.toFieldId(199_999_800)).isEqualTo(999_949);
+ }
- assertThat(StatsUtil.fieldIdForStatsField(-1)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(0)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(200)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(5_000)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(8_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_001)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_201)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_500)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(10_900)).isEqualTo(invalidFieldId);
-
- // stats field IDs at or above the exclusive upper bound are invalid
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(StatsUtil.STATS_SPACE_FIELD_ID_END + 200))
- .isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(Integer.MAX_VALUE)).isEqualTo(invalidFieldId);
-
- // field ID just past MAX_DATA_FIELD_ID is invalid
- assertThat(StatsUtil.statsFieldIdForField(StatsUtil.MAX_DATA_FIELD_ID + 1))
- .isEqualTo(invalidFieldId);
+ @Test
+ public void testToFieldIdWithMetadata() {
+ assertThat(StatsUtil.toFieldId(9_000))
+ .isEqualTo(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId());
+ assertThat(StatsUtil.toFieldId(9_200)).isEqualTo(MetadataColumns.ROW_ID.fieldId());
}
@Test
- public void statsIdsForMetadataColumns() {
- int fieldId = MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
- int statsFieldId = 9_000;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- fieldId = MetadataColumns.ROW_ID.fieldId();
- statsFieldId = 9_200;
- assertThat(StatsUtil.statsFieldIdForField(fieldId)).isEqualTo(statsFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(statsFieldId)).isEqualTo(fieldId);
-
- // reserved metadata fields with IDs below/above the reserved stats range have no stats
- int invalidFieldId = -1;
- assertThat(StatsUtil.fieldIdForStatsField(8_800)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_400)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_600)).isEqualTo(invalidFieldId);
- assertThat(StatsUtil.fieldIdForStatsField(9_800)).isEqualTo(invalidFieldId);
- assertThat(IntStream.range(RESERVED_FIELD_IDS_START, Integer.MAX_VALUE))
- .filteredOn(id -> !StatsUtil.SUPPORTED_METADATA_FIELD_IDS.contains(id))
- .allSatisfy(
- id ->
- assertThat(StatsUtil.statsFieldIdForField(id))
- .as("at pos %s", id)
- .isEqualTo(invalidFieldId));
+ public void testToFieldIdOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.toFieldId(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
}
@Test
- public void contentStatsForSimpleSchema() {
- Types.NestedField intField = required(0, "i", Types.IntegerType.get());
- Types.NestedField floatField = required(2, "f", Types.FloatType.get());
- Types.NestedField stringField = required(4, "s", Types.StringType.get());
- Types.NestedField booleanField = required(6, "b", Types.BooleanType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema = new Schema(intField, floatField, stringField, booleanField, uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(intField, 10000)),
- optional(10400, "f", FieldStatistic.fieldStatsFor(floatField, 10400)),
- optional(10800, "s", FieldStatistic.fieldStatsFor(stringField, 10800)),
- optional(11200, "b", FieldStatistic.fieldStatsFor(booleanField, 11200)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testToFieldIdReservedMetadataRange() {
+ // 9,000 to 10,000 (exclusive) is reserved for metadata column stats
+ assertThatThrownBy(() -> StatsUtil.toFieldId(9_400))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Unsupported metadata stats field ID: 9400");
}
@Test
- public void contentStatsForComplexSchema() {
- Types.NestedField listElement = optional(3, "element", Types.IntegerType.get());
- Types.NestedField structInt = optional(7, "int", Types.IntegerType.get());
- Types.NestedField structString = optional(8, "string", Types.StringType.get());
- Types.NestedField mapKey = required(22, "key", Types.IntegerType.get());
- Types.NestedField mapValue = optional(24, "value", Types.StringType.get());
- Types.NestedField variantField = required(30, "variant", Types.VariantType.get());
- Types.NestedField uuidField = required(StatsUtil.MAX_DATA_FIELD_ID, "u", Types.UUIDType.get());
- Schema schema =
- new Schema(
- required(0, "i", Types.IntegerType.get()),
- required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
- required(
- 6,
- "simple_struct",
- Types.StructType.of(
- optional(7, "int", Types.IntegerType.get()),
- optional(8, "string", Types.StringType.get()))),
- required(
- 20,
- "b",
- Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
- variantField,
- uuidField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(
- 10000,
- "i",
- FieldStatistic.fieldStatsFor(
- required(0, "i", Types.IntegerType.get()), 10000)),
- optional(
- 10600, "list.element", FieldStatistic.fieldStatsFor(listElement, 10600)),
- optional(
- 11400, "simple_struct.int", FieldStatistic.fieldStatsFor(structInt, 11400)),
- optional(
- 11600,
- "simple_struct.string",
- FieldStatistic.fieldStatsFor(structString, 11600)),
- optional(14400, "b.key", FieldStatistic.fieldStatsFor(mapKey, 14400)),
- optional(14800, "b.value", FieldStatistic.fieldStatsFor(mapValue, 14800)),
- optional(16000, "variant", FieldStatistic.fieldStatsFor(variantField, 16000)),
- optional(
- StatsUtil.MAX_DATA_STATS_FIELD_ID,
- "u",
- FieldStatistic.fieldStatsFor(
- uuidField, StatsUtil.MAX_DATA_STATS_FIELD_ID)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ public void testStatOffset() {
+ assertThat(StatsUtil.statOffset(10_000)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(10_201)).isEqualTo(1);
+ assertThat(StatsUtil.statOffset(10_211)).isEqualTo(11);
+ assertThat(StatsUtil.statOffset(10_399)).isEqualTo(199);
+ assertThat(StatsUtil.statOffset(10_400)).isEqualTo(0);
+ assertThat(StatsUtil.statOffset(199_999_999)).isEqualTo(199);
}
@Test
- public void contentStatsChildNamesAreUnique() {
- Schema schema =
- new Schema(
- required(1, "a", Types.StructType.of(required(2, "x", Types.IntegerType.get()))),
- required(3, "b", Types.StructType.of(required(4, "x", Types.IntegerType.get()))));
+ public void testStatOffsetOutsideRange() {
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_000))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000000");
+ assertThatThrownBy(() -> StatsUtil.statOffset(200_000_001))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 200000001");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_800))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8800");
+ assertThatThrownBy(() -> StatsUtil.statOffset(8_801))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Invalid stats field ID: 8801");
+ }
- Types.StructType contentStats = StatsUtil.contentStatsFor(schema).type().asStructType();
+ private static final List<Type> FIXED_WIDTH_TYPES =
+ List.of(
+ Types.BooleanType.get(),
+ Types.IntegerType.get(),
+ Types.LongType.get(),
+ Types.DateType.get(),
+ Types.TimeType.get(),
+ Types.TimestampType.withoutZone(),
+ Types.TimestampType.withZone(),
+ Types.TimestampNanoType.withoutZone(),
+ Types.TimestampNanoType.withZone(),
+ Types.DecimalType.of(9, 2),
+ Types.UUIDType.get(),
+ Types.FixedType.ofLength(16));
+
+ @ParameterizedTest
+ @FieldSource("FIXED_WIDTH_TYPES")
+ public void testFixedWidthPrimitiveStruct(Type type) {
+ // fixed-width, non-floating-point types track bounds (including tight bounds) but have no NaN
+ // count or average value size
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
+ }
- assertThat(contentStats.fields().stream().map(Types.NestedField::name).toList())
- .doesNotHaveDuplicates()
- .containsExactly("a.x", "b.x");
+ private static final List<Type> VARIABLE_WIDTH_TYPES =
+ List.of(Types.StringType.get(), Types.BinaryType.get());
+
+ @ParameterizedTest
+ @FieldSource("VARIABLE_WIDTH_TYPES")
+ public void testVariableWidthPrimitiveStruct(Type type) {
+ // variable-width types track bounds (including tight bounds) and an average value size, but no
+ // NaN count
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", type),
+ Types.NestedField.optional(30_002, "upper_bound", type),
+ Types.NestedField.optional(30_003, "tight_bounds", Types.BooleanType.get()),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
+
+ Types.StructType actual =
+ StatsUtil.fieldStatsStruct(true, type, 30_000, MetricsModes.Full.get());
+
+ assertSameStructure(expected, actual);
}
- @Test
- public void contentStatsSkipsFieldsOutsideStatsRange() {
- Types.NestedField validField = required(0, "i", Types.IntegerType.get());
- Types.NestedField outOfRangeField =
- required(StatsUtil.MAX_DATA_FIELD_ID + 1, "out_of_range", Types.IntegerType.get());
- Schema schema = new Schema(validField, outOfRangeField);
- Schema expectedStatsSchema =
- new Schema(
- optional(
- 146,
- "content_stats",
- Types.StructType.of(
- optional(10000, "i", FieldStatistic.fieldStatsFor(validField, 10000)))));
- Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
- assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ private static final List<Type> GEO_TYPES =
+ List.of(
+ Types.GeometryType.crs84(),
+ Types.GeometryType.of("srid:3857"),
+ Types.GeographyType.crs84(),
+ Types.GeographyType.of("srid:4269"));
+
+ @ParameterizedTest
+ @FieldSource("GEO_TYPES")
+ public void testGeoStruct(Type type) {
+ // geometry and geography use bounding-box structs for their bounds, do not track tight bounds,
+ // and record an average value size
+ Types.StructType lowerBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_010, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_011, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_012, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_013, "m", Types.DoubleType.get()));
+ Types.StructType upperBound =
+ Types.StructType.of(
+ Types.NestedField.required(30_014, "x", Types.DoubleType.get()),
+ Types.NestedField.required(30_015, "y", Types.DoubleType.get()),
+ Types.NestedField.optional(30_016, "z", Types.DoubleType.get()),
+ Types.NestedField.optional(30_017, "m", Types.DoubleType.get()));
+ Types.StructType expected =
+ Types.StructType.of(
+ Types.NestedField.optional(30_001, "lower_bound", lowerBound),
+ Types.NestedField.optional(30_002, "upper_bound", upperBound),
+ Types.NestedField.optional(30_004, "value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_005, "null_value_count", Types.LongType.get()),
+ Types.NestedField.optional(30_007, "avg_value_size_in_bytes", Types.IntegerType.get()));
Review Comment:
Geo types are variable width so I think we should update the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598713082
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
}
- private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
- int fieldId =
- (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN
- + FIRST_SUPPORTED_METADATA_FIELD_ID;
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId) ? fieldId : -1;
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static Types.NestedField contentStatsFor(Schema schema) {
- ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor(schema);
- Types.NestedField result = TypeUtil.visit(schema, visitor);
- if (!visitor.skippedFieldIds.isEmpty()) {
- LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+
+ for (int id : metricsConfig.metricsFieldIds()) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
+
+ int baseId = toBaseId(id);
+ Types.StructType fieldStruct =
+ fieldStatsStruct(field.isOptional(), field.type(), baseId, metricsConfig.columnMode(id));
+
+ if (fieldStruct != null) {
+ fieldStructs.add(optional(baseId, fieldName, fieldStruct));
+ }
}
- return result;
+ return Types.StructType.of(fieldStructs);
}
- private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
- private final Schema tableSchema;
- private final List<Types.NestedField> statsFields = Lists.newArrayList();
- private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+ /**
+ * Produce a schema to read content stats for the given table field IDs.
+ *
+ * @param tableSchema a schema
+ * @param fieldIds an iterable of field IDs to project stats for
+ * @return a content stats struct for a read
+ */
+ public static Types.StructType statsReadSchema(Schema tableSchema, Iterable<Integer> fieldIds) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
- ContentStatsSchemaVisitor(Schema tableSchema) {
- this.tableSchema = tableSchema;
- }
+ for (int id : fieldIds) {
+ String fieldName = idToStatsName.get(id);
+ Types.NestedField field = tableSchema.findField(id);
Review Comment:
Added a check in both places.
For the write path, this causes a failure because it cannot produce a schema to store stats for the requested fields. For the read path, the field is ignored because unavailable stats are okay, they just can't be used to filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589920515
##########
core/src/main/java/org/apache/iceberg/ContentStats.java:
##########
@@ -35,6 +35,12 @@ interface ContentStats extends StructLike {
*/
<T> FieldStats<T> statsFor(int fieldId);
- /** The stats struct holding nested structs with their respective field stats */
- Types.StructType statsStruct();
+ /** Container struct type containing tracked field-level stats structs. */
+ Types.StructType type();
+
+ /** Returns a copy, deep-copying all field stats. */
+ ContentStats copy();
+
+ /** Returns a copy, of only the selected field stats by ID. */
+ ContentStats copy(Set<Integer> fieldIds);
Review Comment:
I don't think so. This is still copying stats, it is just choosing which ones to copy. It is also intended to be consistent with `copyWithStats(...)` in the DataFile API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3603729756
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.lowerBound)
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.upperBound)
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
+ switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> setLowerBound(value);
+ case StatsUtil.UPPER_BOUND_OFFSET -> setUpperBound(value);
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> this.tightBounds = (Boolean) value;
Review Comment:
I guess the same applies for the other fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3598703981
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
Review Comment:
Added a comment to clarify. I don't think the calculation adds much. I stated that this corresponds to stats base ID 200_000_000 that is the end of the stats range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562182767
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.curator.shaded.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ public FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
Review Comment:
Uses offsets to avoid a `switch` by position immediately followed by a `switch` by enum symbol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562178554
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -18,20 +18,18 @@
*/
package org.apache.iceberg;
-import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
-interface FieldStats<T> extends StructLike {
+interface FieldStats<T> {
/** The field ID of the statistic */
int fieldId();
/**
- * The field type of the statistic.
+ * Struct type describing the stats tracked for the field identified by {@link #fieldId()}.
*
- * <p>For geo types (geometry/geography), this returns the bounding box struct type (geo_lower /
- * geo_upper) rather than the column's geometry or geography type, because the type is inferred
- * from the lower/upper bound schema fields.
+ * <p>Note: This type may be a projection of the stats stored in manifest files.
*/
- Type type();
+ Types.StructType type();
Review Comment:
This changed to the actual schema. The lower and upper bounds types do not always match (for geo types) so this is the best way to expose types. It is also more consistent with existing conventions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3562180536
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -46,17 +44,20 @@ interface FieldStats<T> extends StructLike {
boolean tightBounds();
/** The total value count, including null and NaN */
- Long valueCount();
+ long valueCount();
/** The total null value count */
- Long nullValueCount();
+ long nullValueCount();
/** The total NaN value count */
- Long nanValueCount();
+ long nanValueCount();
Review Comment:
These changes are conservative. We may want to widen later, but I want to make a conscious choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3589953254
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
+ switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> setLowerBound(value);
+ case StatsUtil.UPPER_BOUND_OFFSET -> setUpperBound(value);
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> this.tightBounds = (Boolean) value;
+ case StatsUtil.VALUE_COUNT_OFFSET -> this.valueCount = (Long) value;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> this.nullValueCount = (Long) value;
Review Comment:
Internally, we still use `null` to signal unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3590558721
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? ((byte[]) toCopy.lowerBound).clone()
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? ((byte[]) toCopy.upperBound).clone()
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
Review Comment:
got it. let me close this thread to consolidate the discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3603621440
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.lowerBound)
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.upperBound)
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
Review Comment:
should this be called `setFromOffset` because `setOffset` is misleading and indicates that we're setting the offset while we actually set the value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3603675205
##########
core/src/main/java/org/apache/iceberg/FieldStatsStruct.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ByteBuffers;
+
+class FieldStatsStruct<T> implements FieldStats<T>, StructLike, Serializable {
+ private final Types.StructType struct;
+ private final Type boundType;
+ private final int[] posToOffset;
+ private final int fieldId;
+
+ private Object lowerBound = null;
+ private Object upperBound = null;
+ private boolean tightBounds = false;
+ private Long valueCount = null;
+ private Long nullValueCount = null;
+ private Long nanValueCount = null;
+ private Integer avgValueSize = null;
+
+ FieldStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToOffset = posToOffset(struct);
+ this.fieldId = StatsUtil.toFieldId(struct.fields().get(0).fieldId());
+ this.boundType = struct.fieldType("lower_bound");
+ }
+
+ FieldStatsStruct(
+ Types.StructType struct,
+ T lowerBound,
+ T upperBound,
+ boolean tightBounds,
+ long valueCount,
+ long nullValueCount,
+ long nanValueCount,
+ Integer avgValueSize) {
+ this(struct);
+ setLowerBound(lowerBound);
+ setUpperBound(upperBound);
+ this.tightBounds = tightBounds;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ }
+
+ private FieldStatsStruct(FieldStatsStruct<T> toCopy) {
+ this(toCopy.struct);
+ // bounds are stored using the internal representation, which is a byte array for binary types
+ this.lowerBound =
+ toCopy.lowerBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.lowerBound)
+ : toCopy.lowerBound;
+ this.upperBound =
+ toCopy.upperBound instanceof byte[]
+ ? copyOf((byte[]) toCopy.upperBound)
+ : toCopy.upperBound;
+ this.tightBounds = toCopy.tightBounds;
+ this.valueCount = toCopy.valueCount;
+ this.nullValueCount = toCopy.nullValueCount;
+ this.nanValueCount = toCopy.nanValueCount;
+ this.avgValueSize = toCopy.avgValueSize;
+ }
+
+ void fromFieldMetrics(FieldMetrics<T> fieldMetrics) {
+ Preconditions.checkArgument(
+ fieldMetrics.id() == fieldId,
+ "Cannot store stats for field ID: %s (expected %s)",
+ fieldMetrics.id(),
+ fieldId);
+
+ setLowerBound(fieldMetrics.lowerBound());
+ setUpperBound(fieldMetrics.upperBound());
+ this.tightBounds = false;
+ this.valueCount = fieldMetrics.valueCount();
+ this.nullValueCount = fieldMetrics.nullValueCount() < 0 ? null : fieldMetrics.nullValueCount();
+ this.nanValueCount = fieldMetrics.nanValueCount() < 0 ? null : fieldMetrics.nanValueCount();
+ this.avgValueSize = null;
+ }
+
+ private boolean isBinary() {
+ return boundType != null
+ && (boundType.typeId() == Type.TypeID.FIXED || boundType.typeId() == Type.TypeID.BINARY);
+ }
+
+ private void setLowerBound(Object lowerBound) {
+ this.lowerBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) lowerBound) : lowerBound;
+ }
+
+ private void setUpperBound(Object upperBound) {
+ this.upperBound = isBinary() ? ByteBuffers.toByteArray((ByteBuffer) upperBound) : upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Types.StructType type() {
+ return struct;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T lowerBound() {
+ return (T)
+ (lowerBound != null && isBinary() ? ByteBuffer.wrap((byte[]) lowerBound) : lowerBound);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T upperBound() {
+ return (T)
+ (upperBound != null && isBinary() ? ByteBuffer.wrap((byte[]) upperBound) : upperBound);
+ }
+
+ @Override
+ public boolean tightBounds() {
+ return tightBounds;
+ }
+
+ @Override
+ public long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSizeInBytes() {
+ return avgValueSize;
+ }
+
+ @Override
+ public int size() {
+ return struct.fields().size();
+ }
+
+ private Object getOffset(int offset) {
+ return switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();
+ case StatsUtil.UPPER_BOUND_OFFSET -> upperBound();
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> tightBounds;
+ case StatsUtil.VALUE_COUNT_OFFSET -> valueCount;
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET -> nullValueCount;
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET -> nanValueCount;
+ case StatsUtil.AVG_VALUE_SIZE_OFFSET -> avgValueSize;
+ default -> throw new UnsupportedOperationException("Unsupported stats offset: " + offset);
+ };
+ }
+
+ @Override
+ public <C> C get(int pos, Class<C> javaClass) {
+ return javaClass.cast(getOffset(posToOffset[pos]));
+ }
+
+ private void setOffset(int offset, Object value) {
+ switch (offset) {
+ case StatsUtil.LOWER_BOUND_OFFSET -> setLowerBound(value);
+ case StatsUtil.UPPER_BOUND_OFFSET -> setUpperBound(value);
+ case StatsUtil.TIGHT_BOUNDS_OFFSET -> this.tightBounds = (Boolean) value;
Review Comment:
wouldn't this fail with a NPE when a `null` is passed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3591825068
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
Review Comment:
Do something like Iterables.transform(fieldIds, StatsUtil::toBaseId)) to get to the right ID space?
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
Review Comment:
I _think_ this is mixing up two ID spaces. `fieldIds` are table field IDs, which makes sense because we are relying on it to build `idToFieldStats`. But the `toCopy.struct` is keyed off the stats field IDs. As a result, I think `TypeUtil.select(toCopy.struct, fieldIds)` might be returning an empty struct.
##########
core/src/main/java/org/apache/iceberg/ContentStatsStruct.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+class ContentStatsStruct implements ContentStats, StructLike, Serializable {
+ private final Map<Integer, FieldStats<?>> idToFieldStats = Maps.newHashMap();
+ private final Types.StructType struct;
+ private final int[] posToId;
+
+ ContentStatsStruct(Types.StructType struct) {
+ this.struct = struct;
+ this.posToId = posToId(struct);
+ }
+
+ private ContentStatsStruct(ContentStatsStruct toCopy, Set<Integer> fieldIds) {
+ this(fieldIds != null ? TypeUtil.select(toCopy.struct, fieldIds) : toCopy.struct);
+ if (fieldIds != null) {
+ for (int fieldId : fieldIds) {
+ idToFieldStats.put(fieldId, toCopy.idToFieldStats.get(fieldId).copy());
Review Comment:
Won't this cause a NPE if the requested ID has no stored stats? Should we skip the IDs rather than dereference them?
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
}
- private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
- int fieldId =
- (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN
- + FIRST_SUPPORTED_METADATA_FIELD_ID;
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId) ? fieldId : -1;
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static Types.NestedField contentStatsFor(Schema schema) {
- ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor(schema);
- Types.NestedField result = TypeUtil.visit(schema, visitor);
- if (!visitor.skippedFieldIds.isEmpty()) {
- LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+
+ for (int id : metricsConfig.metricsFieldIds()) {
Review Comment:
The field order is non-determinisitc because we are iterating over a hash map. Should we sort by IDs? There is no correctness issue here.
##########
core/src/test/java/org/apache/iceberg/TestContentStatsStruct.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.TestHelpers.RoundTripSerializer;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+import org.mockito.Mockito;
+
+public class TestContentStatsStruct {
+ private static final Schema SCHEMA =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.StringType.get()),
+ optional(3, "score", Types.DoubleType.get()));
+
+ private static final Types.StructType CONTENT_STATS_STRUCT =
+ StatsUtil.statsReadSchema(SCHEMA, ImmutableList.of(1, 2, 3));
+ private static final Types.StructType UNKNOWN_FIELD_STATS_STRUCT =
+ StatsUtil.fieldStatsStruct(false, Types.IntegerType.get(), 10_800, MetricsModes.Full.get());
+
+ private static final FieldStats<Long> ID_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("id").type().asStructType(), 0L, 25L, true, 26L, 0L, 0L, null);
+ private static final FieldStats<String> DATA_STATS =
+ new FieldStatsStruct<>(
+ CONTENT_STATS_STRUCT.field("data").type().asStructType(),
+ "a",
+ "z",
+ true,
+ 26L,
+ 0L,
+ 0L,
+ null);
+
+ @Test
+ public void testEmptyContentStats() {
+ ContentStats stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.statsFor(1)).isNull();
+ assertThat(stats.statsFor(2)).isNull();
+ assertThat(stats.statsFor(3)).isNull();
+ assertThat(stats.statsFor(4)).as("Should ignore unknown field IDs").isNull();
+ }
+
+ @Test
+ public void testSetStats() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.statsFor(1)).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(2)).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(3)).isNull();
+ }
+
+ @Test
+ public void testSetStatsWrongId() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThatThrownBy(() -> stats.setStats(2, ID_STATS))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Mismatched field stats for ID 2: actual ID 1");
+ }
+
+ @Test
+ public void testSetStatsUnknownField() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ FieldStats<Integer> fieldStats =
+ new FieldStatsStruct<>(UNKNOWN_FIELD_STATS_STRUCT, 0, 10, false, 8, 3, 0, null);
+
+ assertThatThrownBy(() -> stats.setStats(4, fieldStats))
+ .hasMessage("Cannot set stats for unknown field ID: 4")
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
+ public void testGetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType contentStatsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(contentStatsStruct);
+
+ stats.setStats(1, ID_STATS);
+ stats.setStats(2, DATA_STATS);
+
+ assertThat(stats.get(0, FieldStats.class)).isEqualTo(DATA_STATS);
+ assertThat(stats.get(1, FieldStats.class)).isNull();
+ assertThat(stats.get(2, FieldStats.class)).isEqualTo(ID_STATS);
+ }
+
+ @Test
+ public void testSetByPosition() {
+ // the content stats struct with a known field order
+ Types.StructType statsStruct =
+ Types.StructType.of(
+ CONTENT_STATS_STRUCT.field("data"),
+ CONTENT_STATS_STRUCT.field("score"),
+ CONTENT_STATS_STRUCT.field("id"));
+
+ ContentStatsStruct stats = new ContentStatsStruct(statsStruct);
+
+ stats.set(1, null);
+ stats.set(2, ID_STATS);
+ stats.set(0, DATA_STATS);
+
+ assertThat(stats.statsFor(SCHEMA.findField("id").fieldId())).isEqualTo(ID_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("data").fieldId())).isEqualTo(DATA_STATS);
+ assertThat(stats.statsFor(SCHEMA.findField("score").fieldId())).isNull();
+ }
+
+ @Test
+ public void testSize() {
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+
+ assertThat(stats.size()).isEqualTo(3);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ ContentStats copy = stats.copy();
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(stats.type());
+
+ // each field stats is deep-copied using its own copy() method
+ assertThat(copy.statsFor(1)).isSameAs(idStatsCopy);
+ assertThat(copy.statsFor(2)).isSameAs(dataStatsCopy);
+ assertThat(copy.statsFor(3)).isNull();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testFilteredCopy() {
+ FieldStats<Long> idStats = Mockito.mock(FieldStats.class);
+ FieldStats<Long> idStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(idStats.fieldId()).thenReturn(1);
+ Mockito.when(idStats.copy()).thenReturn(idStatsCopy);
+
+ FieldStats<String> dataStats = Mockito.mock(FieldStats.class);
+ FieldStats<String> dataStatsCopy = Mockito.mock(FieldStats.class);
+ Mockito.when(dataStats.fieldId()).thenReturn(2);
+ Mockito.when(dataStats.copy()).thenReturn(dataStatsCopy);
+
+ ContentStatsStruct stats = new ContentStatsStruct(CONTENT_STATS_STRUCT);
+ stats.setStats(1, idStats);
+ stats.setStats(2, dataStats);
+
+ // copy just the stats for field ID 2
+ ContentStats copy = stats.copy(ImmutableSet.of(2));
+
+ assertThat(copy).isInstanceOf(ContentStatsStruct.class).isNotSameAs(stats);
+ assertThat(copy.type()).isEqualTo(TypeUtil.select(stats.type(), ImmutableSet.of(1)));
+
Review Comment:
Yes, I think there is a bug in the code, so the test is passing vacuously. I left a comment in ContentsStatsStruct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#discussion_r3599178369
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -19,171 +19,291 @@
package org.apache.iceberg;
import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
-import java.util.Collections;
-import java.util.Comparator;
import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
class StatsUtil {
- private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
- static final Set<Integer> SUPPORTED_METADATA_FIELD_IDS =
- ImmutableSet.of(
- MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), MetadataColumns.ROW_ID.fieldId());
- private static final int FIRST_SUPPORTED_METADATA_FIELD_ID =
- Collections.min(SUPPORTED_METADATA_FIELD_IDS);
- static final int NUM_SUPPORTED_STATS_PER_COLUMN = 200;
- static final int STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS = 9_000;
- static final int STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS = 10_000;
- // exclusive upper bound of the stats field ID range reserved for content_stats
- static final int STATS_SPACE_FIELD_ID_END = 200_000_000;
- static final int MAX_DATA_STATS_FIELD_ID =
- STATS_SPACE_FIELD_ID_END - NUM_SUPPORTED_STATS_PER_COLUMN;
- // the max data field ID whose stats struct fits within the reserved range
- static final int MAX_DATA_FIELD_ID =
- (MAX_DATA_STATS_FIELD_ID - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
-
private StatsUtil() {}
- public static int statsFieldIdForField(int fieldId) {
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId)
- ? statsFieldIdForReservedField(fieldId)
- : statsFieldIdForDataField(fieldId);
- }
+ private static final int NUM_RESERVED_FIELD_STATS_IDS = 200;
+ private static final int METADATA_STATS_RANGE_START = 9_000;
+ private static final int CONTENT_STATS_RANGE_START = 10_000;
+ private static final int CONTENT_STATS_RANGE_END = 200_000_000; // exclusive
+
+ private static final int DATA_FIELD_ID_START = 0;
+ private static final int DATA_FIELD_ID_END = 999_950; // exclusive
+
+ private static final int LAST_UPDATED_SEQ_NUM_BASE_ID = 9_000;
+ private static final int ROW_ID_BASE_ID = 9_200;
+
+ // Offsets used for individual stats columns
+ static final int LOWER_BOUND_OFFSET = 1;
+ static final int UPPER_BOUND_OFFSET = 2;
+ static final int TIGHT_BOUNDS_OFFSET = 3;
+ static final int VALUE_COUNT_OFFSET = 4;
+ static final int NULL_VALUE_COUNT_OFFSET = 5;
+ static final int NAN_VALUE_COUNT_OFFSET = 6;
+ static final int AVG_VALUE_SIZE_OFFSET = 7;
+
+ // Offsets used within geo_lower struct
+ private static final int GEO_LOWER_X_OFFSET = 10;
+ private static final int GEO_LOWER_Y_OFFSET = 11;
+ private static final int GEO_LOWER_Z_OFFSET = 12;
+ private static final int GEO_LOWER_M_OFFSET = 13;
+
+ // Offsets used within geo_upper struct
+ private static final int GEO_UPPER_X_OFFSET = 14;
+ private static final int GEO_UPPER_Y_OFFSET = 15;
+ private static final int GEO_UPPER_Z_OFFSET = 16;
+ private static final int GEO_UPPER_M_OFFSET = 17;
- private static int statsFieldIdForDataField(int fieldId) {
- if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
- return -1;
+ private static final Map<Integer, Integer> METADATA_ID_TO_BASE_ID =
+ ImmutableMap.of(
+ MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId(), LAST_UPDATED_SEQ_NUM_BASE_ID,
+ MetadataColumns.ROW_ID.fieldId(), ROW_ID_BASE_ID);
+
+ /**
+ * Return the base ID of the stats struct for the given field ID, or -1 if the ID is out of range.
+ *
+ * @param fieldId a table field ID
+ * @return the base ID for a field stats struct, or -1 if stats cannot be stored
+ */
+ @VisibleForTesting
+ static int toBaseId(int fieldId) {
+ if (fieldId >= DATA_FIELD_ID_START && fieldId < DATA_FIELD_ID_END) {
+ return (fieldId * NUM_RESERVED_FIELD_STATS_IDS) + CONTENT_STATS_RANGE_START;
}
- return STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS + (NUM_SUPPORTED_STATS_PER_COLUMN * fieldId);
+ return METADATA_ID_TO_BASE_ID.getOrDefault(fieldId, -1);
}
- private static int statsFieldIdForReservedField(int fieldId) {
- return STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- + (NUM_SUPPORTED_STATS_PER_COLUMN * (fieldId - FIRST_SUPPORTED_METADATA_FIELD_ID));
- }
+ /**
+ * Return the field ID corresponding to the stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return ID of the corresponding table field
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int toFieldId(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
- public static int fieldIdForStatsField(int statsFieldId) {
- if (statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS
- || statsFieldId >= STATS_SPACE_FIELD_ID_END
- || statsFieldId % NUM_SUPPORTED_STATS_PER_COLUMN != 0) {
- return -1;
+ if (statId < CONTENT_STATS_RANGE_START) {
+ if (inBaseIdRange(LAST_UPDATED_SEQ_NUM_BASE_ID, statId)) {
+ return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
+ } else if (inBaseIdRange(ROW_ID_BASE_ID, statId)) {
+ return MetadataColumns.ROW_ID.fieldId();
+ } else {
+ throw new UnsupportedOperationException("Unsupported metadata stats field ID: " + statId);
+ }
}
- return statsFieldId < STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS
- ? fieldIdForStatsFieldFromReservedField(statsFieldId)
- : fieldIdForStatsFieldFromDataField(statsFieldId);
+ return (statId - CONTENT_STATS_RANGE_START) / NUM_RESERVED_FIELD_STATS_IDS;
}
- private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
- return (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_DATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN;
+ /**
+ * Return the stats offset of a stats field ID.
+ *
+ * @param statId the field ID of a field stats struct or field within a stats struct
+ * @return offset that identifies the stored metric, or 0 for a stats struct's ID
+ * @throws IllegalArgumentException if the stats ID is not valid
+ * @throws UnsupportedOperationException if the stats ID is for an unsupported metadata field
+ */
+ static int statOffset(int statId) {
+ Preconditions.checkArgument(isValidStatId(statId), "Invalid stats field ID: %s", statId);
+
+ return statId % 200;
}
- private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
- int fieldId =
- (statsFieldId - STATS_SPACE_FIELD_ID_START_FOR_METADATA_FIELDS)
- / NUM_SUPPORTED_STATS_PER_COLUMN
- + FIRST_SUPPORTED_METADATA_FIELD_ID;
- return SUPPORTED_METADATA_FIELD_IDS.contains(fieldId) ? fieldId : -1;
+ public static Types.NestedField contentStatsField(Types.StructType contentStats) {
+ return optional(146, "content_stats", contentStats);
}
- public static Types.NestedField contentStatsFor(Schema schema) {
- ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor(schema);
- Types.NestedField result = TypeUtil.visit(schema, visitor);
- if (!visitor.skippedFieldIds.isEmpty()) {
- LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ public static Types.StructType statsWriteSchema(Schema tableSchema, MetricsConfig metricsConfig) {
+ Map<Integer, String> idToStatsName = TypeUtil.indexStatsNames(tableSchema.asStruct());
+ List<Types.NestedField> fieldStructs = Lists.newArrayList();
+
+ for (int id : metricsConfig.metricsFieldIds()) {
Review Comment:
I thought about this and wanted to preserve the schema order. At the time, it seemed like a good idea to use the order from the input (in this case `metricsFieldIds`) rather than sorting by field ID because sorting would erase the schema order (this is why the read path accepts `Iterable`). However, you're right that `metricsFieldIds` is unordered.
I think what we want is to preserve the schema order, but I'd like to do it in a follow up since it would probably make more changes to `MetricsConfig` and I tried to separate those changes out (into #17022).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #17159:
URL: https://github.com/apache/iceberg/pull/17159#issuecomment-5006807033
Thanks for the reviews! I'm going to merge this to unblock other changes and integration with the readers. We can still follow up on a few things, like whether `tightBounds` is required or optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Refactor ContentStats and FieldStats [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #17159:
URL: https://github.com/apache/iceberg/pull/17159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org