You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nastra (via GitHub)" <gi...@apache.org> on 2025/08/27 15:27:31 UTC
[PR] API, Core: Introduce classes for content stats [iceberg]
nastra opened a new pull request, #13933:
URL: https://github.com/apache/iceberg/pull/13933
This introduces just the base classes that are needed for content stats and is extracted from https://github.com/apache/iceberg/pull/13694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2374650254
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
Yep, we can only change this before the next release and my assumption here is that it might be a while until we do another release with v4 stuff, so we should have enough time to finalize this API in case we want to make this generic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2310977868
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
Review Comment:
nit on the name, statsIdForField?
I may be being paranoid here but I feel like we are putting a lot of math in here and at least to me it's obscuring things a bit.
Maybe instead we could do
statsIdForField (fieldId) {
Precondition (fieldID >= 0)
return (fieldId >= RESEREVED_FIELD_ID_START) ?
statsIdForReservedField(fieldId) :
statsIdForDataField(fieldId)
}
statsIdForDataField(fieldId) {
statsId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) id;
//overflow conditions specific to data field
}
statsIdForReservedField(fieldId) {
statId = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldID)
// Overflow check for ReservedFields
}
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
Review Comment:
nit on the name, statsIdForField?
I may be being paranoid here but I feel like we are putting a lot of math in here and at least to me it's obscuring things a bit.
Maybe instead we could do
```
statsIdForField (fieldId) {
Precondition (fieldID >= 0)
return (fieldId >= RESEREVED_FIELD_ID_START) ?
statsIdForReservedField(fieldId) :
statsIdForDataField(fieldId)
}
statsIdForDataField(fieldId) {
statsId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) id;
//overflow conditions specific to data field
}
statsIdForReservedField(fieldId) {
statId = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldID)
// Overflow check for ReservedFields
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2482163292
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
Review Comment:
`STATS_SPACE_FIELD_ID_START`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349170460
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
Review Comment:
I initially made this a typed class, but the type information didn't actually help with anything and didn't make things clearer. You were still able to e.g. create a `FieldStats<X> stats = BaseFieldStats.builder()....type(Y).build()` where the underlying `type` doesn't match the type parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2457685329
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
Review Comment:
Shouldn't we only produce this for types that have `NaN`? I think we want to be type specific with respect to the schema we produce (especially if we add other stats for more complex types like geo)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349063537
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
Review Comment:
I've renamed this to `statsFieldIdForField(...)` and I agree that extracting the math out into their own methods makes things easier to reason about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2421574548
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
+
+ /** A list of all the {@link FieldStats} */
+ List<FieldStats<?>> fieldStats();
Review Comment:
I feel like we're going to want to be able to grab the stats for a field id (based on the schema's field id). So maybe something like `FieldStats<?> fieldsStats(int fieldId)`. We could have lots of stats, so traversing all the stats to find the ones we care about could be expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409840426
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
Throwing here would result in an exception when reading a manifest and I don't think we would want to do that because I think it's better to not fail in general. Instead it's better to indicate that there are no stats available for field ID X
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2371981240
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
sorry, I forgot that I moved all of the stats schema code, so the warn reporting wouldn't happen in a separate PR but in this one. Basically if we would want to log a warn then this would happen in https://github.com/apache/iceberg/pull/13933/files#diff-5467d1f9b780966e1d3dd2b3c32dc745a3988038046d8f3c1e90d739223a0aaaR184 and this is the place that uses the `-1` but I'm still undecided whether it's a good idea to actually do that warn or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2372279293
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
I'm ok with keeping unparameterized for now.
OTOH we should remember that we can only change this until the next release. Once the unparameterized version is released then it would be a pain if we want to change it generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#issuecomment-3520312080
thanks for the reviews @pvary, @stevenzwu, @danielcweeks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311069516
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void statsIdsOverflowForTableColumns() {
+ for (int i = 0; i < 100; i++) {
+ int id =
+ ThreadLocalRandom.current()
+ .nextInt(StatsUtil.METADATA_SPACE_FIELD_ID_START, StatsUtil.RESERVED_FIELD_IDS_START);
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = -1;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(id)).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ public void statsIdsForReservedColumns() {
+ int offset = 0;
+ for (int id = StatsUtil.RESERVED_FIELD_IDS_START; id < Integer.MAX_VALUE; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.METADATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset = offset + StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void contentStatsForSimpleSchema() {
+ Schema schema =
+ new Schema(
+ required(0, "i", Types.IntegerType.get()),
+ required(2, "f", Types.FloatType.get()),
+ required(4, "s", Types.StringType.get()),
+ required(6, "b", Types.BooleanType.get()),
+ required(250_000, "u", Types.UUIDType.get()));
+ Schema expectedStatsSchema =
+ new Schema(
+ optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ optional(10000, "0", fieldStatsFor(Types.IntegerType.get(), 10001)),
+ optional(10400, "2", fieldStatsFor(Types.FloatType.get(), 10401)),
+ optional(10800, "4", fieldStatsFor(Types.StringType.get(), 10801)),
+ optional(11200, "6", fieldStatsFor(Types.BooleanType.get(), 11201)),
+ optional(50010000, "250000", fieldStatsFor(Types.UUIDType.get(), 50010001)))));
+ Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
+ assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ }
+
+ @Test
+ public void contentStatsForComplexSchema() {
+ Schema schema =
+ new Schema(
+ required(0, "i", Types.IntegerType.get()),
+ required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
+ required(
+ 6,
+ "simple_struct",
+ Types.StructType.of(
+ optional(7, "int", Types.IntegerType.get()),
+ optional(8, "string", Types.StringType.get()))),
+ required(
+ 20,
+ "b",
+ Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
+ required(30, "variant", Types.VariantType.get()),
+ required(250_000, "u", Types.UUIDType.get()));
+ Schema expectedStatsSchema =
+ new Schema(
+ optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ optional(10000, "0", fieldStatsFor(Types.IntegerType.get(), 10001)),
+ optional(10600, "3", fieldStatsFor(Types.IntegerType.get(), 10601)),
+ optional(11400, "7", fieldStatsFor(Types.IntegerType.get(), 11401)),
+ optional(11600, "8", fieldStatsFor(Types.StringType.get(), 11601)),
+ optional(14400, "22", fieldStatsFor(Types.IntegerType.get(), 14401)),
+ optional(14800, "24", fieldStatsFor(Types.StringType.get(), 14801)),
+ optional(50010000, "250000", fieldStatsFor(Types.UUIDType.get(), 50010001)))));
+ Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
+ assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ }
+
+ private Type fieldStatsFor(Type type, int id) {
+ int fieldId = id;
Review Comment:
I'm generally worried about test code which mimics the implementation. I think ideally we would have some code for reading the stats out of the structure and we could check they matched the values we put in. That may be outside the scope of this file though ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311016355
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
Review Comment:
This constant is defined somewhere too right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2387225424
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats<T> implements FieldStats<T>, StructLike, Serializable {
+ private final transient int fieldId;
+ private final transient Type type;
+ private final Long columnSize;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final T lowerBound;
+ private final T upperBound;
+
+ BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long columnSize,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ T lowerBound,
+ T upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.columnSize = columnSize;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
Review Comment:
yeah good point, that makes sense: I've added that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2465189666
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
+
+ /** A list of all the {@link FieldStats} */
+ List<FieldStats<?>> fieldStats();
Review Comment:
I went ahead and added `statsFor(fieldId)` as part of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368787155
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ return javaClass.cast(fieldStats.get(pos));
+ }
+
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ BaseFieldStats stat = (BaseFieldStats) fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("column_size")) {
+ builder.columnSize((Long) record.getField("column_size"));
+ }
+ if (null != record.getField("value_count")) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409306627
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
Review Comment:
nit: id -> startingFieldId?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2407938687
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
Review Comment:
throws an IllegalArgumentException for the condition of `fieldId < 0`?
The other two conditions look good to me
* `fieldId > MAX_DATA_FIELD_ID` (number of columns over 1M), we just return `-1` to indicate no column stats should be written?
* `statsFieldId < 0`: ID overflow. technically this is redundant as the `fieldId > MAX_DATA_FIELD_ID` should cover it too.
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
Review Comment:
nit: statsFieldStartingIdForDataField(init fieldId)? this is the starting/base id
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
Review Comment:
variant is also a variable length type?
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
+ // don't overflow and don't overlap with the metadata ID range
+ if (fieldId >= 0) {
+ Types.StructType structType = contentStatsFor(field.type(), fieldId + 1);
Review Comment:
why `+ 1` here? shouldn't it be `statsFieldStartingId` directly?
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
Review Comment:
since reserved metadata fields are Iceberg internal code, we probably should fail in this case?
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
should we explain the reason for skipping? stats are only allowed for field id up to 1M.
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
while we can get ID overflow for the direction of mapping field id to stats field id, we probably should throw an `IllegalArgumentException` for the direction if `statsFieldId` doesn't conform to the expected validations.
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
Review Comment:
nit: fieldId -> statsFieldStartingId. also don't need `StatsUtils.` since it is in the same class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413814480
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
Review Comment:
I'll add a comment to make this clearer, thanks for bringing this up. Also yes, `< 0` is an artificial case that won't happen usually and I only added this for testing. After re-checking the entire codebase on implementations of `StructLike#get`, the only place that actually does a positional check is `PartitionData` [here](https://github.com/apache/iceberg/blob/fb02bd2d713c9bef8b8ccee334f6d65087c344b2/core/src/main/java/org/apache/iceberg/PartitionData.java#L133-L135)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413834174
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
Review Comment:
see my other [comment](https://github.com/apache/iceberg/pull/13933#discussion_r2413779883) on this. I don't think it helps by introducing a new util method, since we don't have any other use cases for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311024896
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
Review Comment:
Feel free to ignore, but it's weird to me that we have keyResult and valueResult already as arguments and this method just takes map and builds a new list from them
```java
private List<NestedField> lazyFieldList() {
if (fields == null) {
this.fields = ImmutableList.of(keyField, valueField);
}
return fields;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311032060
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
Review Comment:
Like the below example, this just wraps ElementField in a list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2485539869
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
Review Comment:
I've slightly renamed the constants and also added comments everywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349067212
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
Review Comment:
I don't think we want to break here. Instead I believe it's better to just not have any stats when the ID overflows for whatever reason (which is why I'm returning/using the -1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349193825
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
Review Comment:
I've seem other impls across the codebase that basically do a similar thing, so I've been just aligning the impl here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#issuecomment-3318998314
What do we think about the error handling? I see that in many places we just return -1.
I think we would like to highlight the cases when we have unexpected results. We might decide to keep writing out data when the stats are facing unexpected input, but I think minimally we need to log warn messages.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368544403
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
I commented about this in https://github.com/apache/iceberg/pull/13933#discussion_r2349170460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2371950288
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
So we will do the schema check in a single place, and log a warn on any field where the id is invalid?
Are there any `legitimate` cases where the `-1` is used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2387753619
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ return javaClass.cast(fieldStats.get(pos));
Review Comment:
Do we want to check the types here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409887365
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
Review Comment:
no, we need to return null here. This has to do with the fact of how Avro reads the stats schema and how it writes stats entries. For example if you have a table with two columns, you will end up with a stats schema that holds two nested structs as shown below (one for each table column):
```
10200: 1: optional struct<10201: column_size: optional long (Total size on disk), 10202: value_count: optional long (Total value count, including null and NaN), 10203: null_value_count: optional long (Total null value count), 10204: nan_value_count: optional long (Total NaN value count), 10205: lower_bound: optional int (Lower bound), 10206: upper_bound: optional int (Upper bound)>
```
```
10400: 2: optional struct<10401: column_size: optional long (Total size on disk), 10402: value_count: optional long (Total value count, including null and NaN), 10403: null_value_count: optional long (Total null value count), 10404: nan_value_count: optional long (Total NaN value count), 10405: lower_bound: optional string (Lower bound), 10406: upper_bound: optional string (Upper bound)>
```
Now if Avro only writes stats for the first column (pos = 0), it will later still evaluate the stats schema and look at pos = 1, which would result in an exception, because we don't hold any stats here for this particular column and that's why we return null here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409798243
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
Review Comment:
we're already making sure that lower/upper bound have the same type when stats are created, so checking it here again seems to be unnecessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409342318
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
Review Comment:
nit on error msg: `Field stats must be a struct type: %s`
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
Review Comment:
nit: id -> startingFieldId?
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
Review Comment:
nested ternary operator is a bit difficult to read. change it to a method that uses if-else?
maybe we should also check that if both lower_bound and upper_bound are not null, they should have the same type.
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
+ // don't overflow and don't overlap with the metadata ID range
+ if (fieldId >= 0) {
+ Types.StructType structType = contentStatsFor(field.type(), fieldId + 1);
Review Comment:
oh. I think I understand now. the struct field itself has `fieldId`. the child stats field start with offset 0. that's why we need the `+ 1` here. might be a bit more natural to have offset starting at 1 and remove the `+ 1` here. but it could be just my personal taste.
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
+ // don't overflow and don't overlap with the metadata ID range
+ if (fieldId >= 0) {
+ Types.StructType structType = contentStatsFor(field.type(), fieldId + 1);
Review Comment:
nm. offset starting at 0 works better with the get by position for StructLike
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
Review Comment:
we are simulating a StructLike with a list? seems a little odd to me. does this class need to be `StructLike`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409398335
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
Review Comment:
if `pos` is out of range, should we fail here?
also related to the comment below, I don't get the use case of get by position for StructLike here. `Class<T> javaClass` should always be a `FieldStats`.
I can probably see a reason for a getter by field id
```
public FieldStats<?> get(int fieldId)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311034005
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
Review Comment:
I guess it makes the code all look more similar ... so maybe it's good to keep these calls to field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2375887762
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import java.util.StringJoiner;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats implements FieldStats, StructLike, Serializable {
+ private final transient int fieldId;
+ private final transient Type type;
+ private final Long columnSize;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final Object lowerBound;
+ private final Object upperBound;
+
+ BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long columnSize,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ Object lowerBound,
+ Object upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.columnSize = columnSize;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Type type() {
+ return type;
+ }
+
+ @Override
+ public Long columnSize() {
+ return columnSize;
+ }
+
+ @Override
+ public Long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public Long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public Long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Object lowerBound() {
+ return lowerBound;
+ }
+
+ @Override
+ public Object upperBound() {
+ return upperBound;
+ }
+
+ @Override
+ public int size() {
+ return 6;
+ }
+
+ @Override
+ public Object get(int pos, Class javaClass) {
+ switch (pos) {
+ case StatsUtil.COLUMN_SIZE_OFFSET:
+ return javaClass.cast(columnSize);
+ case StatsUtil.VALUE_COUNT_OFFSET:
+ return javaClass.cast(valueCount);
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET:
+ return javaClass.cast(nullValueCount);
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET:
+ return javaClass.cast(nanValueCount);
+ case StatsUtil.LOWER_BOUND_OFFSET:
+ return javaClass.cast(lowerBound);
+ case StatsUtil.UPPER_BOUND_OFFSET:
+ return javaClass.cast(upperBound);
+ default:
+ throw new UnsupportedOperationException("Unknown field ordinal: " + pos);
+ }
+ }
+
+ @Override
+ public void set(int pos, Object value) {
+ throw new UnsupportedOperationException("set() not supported");
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", BaseFieldStats.class.getSimpleName() + "[", "]")
Review Comment:
nit: Maybe use MoreObjects here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368358032
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ return javaClass.cast(fieldStats.get(pos));
+ }
+
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ BaseFieldStats stat = (BaseFieldStats) fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("column_size")) {
+ builder.columnSize((Long) record.getField("column_size"));
+ }
+ if (null != record.getField("value_count")) {
+ builder.valueCount((Long) record.getField("value_count"));
+ }
+ if (null != record.getField("nan_value_count")) {
+ builder.nanValueCount((Long) record.getField("nan_value_count"));
+ }
+ if (null != record.getField("null_value_count")) {
+ builder.nullValueCount((Long) record.getField("null_value_count"));
+ }
+ if (null != record.getField("lower_bound")) {
+ Object lowerBound = record.getField("lower_bound");
+ if (null != type) {
+ builder.lowerBound(type.typeId().javaClass().cast(lowerBound));
+ }
+ }
+ if (null != record.getField("upper_bound")) {
+ Object upperBound = record.getField("upper_bound");
+ if (null != type) {
+ builder.upperBound(type.typeId().javaClass().cast(upperBound));
+ }
+ }
+
+ BaseFieldStats newStat = builder.build();
+ fieldStats.set(pos, newStat);
+ } else {
+ fieldStats.set(pos, (FieldStats) value);
+ }
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", BaseContentStats.class.getSimpleName() + "[", "]")
Review Comment:
Any specific reason not to use `MoreObjects.toStringHelper(this)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2391984595
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
Review Comment:
nit: add a test for this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2399582858
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
Review Comment:
does this interface needs to expend `StructLike`?
I assume `content` means that this is also used for equality delete files (in addition to data files)? is `ContentFileStats` more informative?
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats<T> extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Integer avgValueSize();
Review Comment:
nit: maybe add Javadoc for this and next method? especially explain they are for variable length fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2408376408
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
+ visitor.skippedFieldIds,
+ schema.asStruct());
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
Review Comment:
nit: fieldId -> statsFieldStartingId. also don't need `StatsUtils.` since it is in the same class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2408387033
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
Review Comment:
nit: statsFieldStartingIdForDataField(init fieldId)? this is the starting/base id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409783794
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
Review Comment:
yes this needs to be a `StructLike` and this part has todo with how things are read by Avro. Things will get clearer when looking at the PR that contains the full POC work and verifies that everything works with Avro, which is https://github.com/apache/iceberg/pull/13694 but we can't currently use that one because the way we map the v4 structure is still WIP and we haven't defined yet where the `content_stats` object will be hooked into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2465189666
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
+
+ /** A list of all the {@link FieldStats} */
+ List<FieldStats<?>> fieldStats();
Review Comment:
I went ahead and added `statsFor(fieldID)` as part of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2412032315
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
Review Comment:
check the pos value for the valid range of fieldStats list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413839757
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
Review Comment:
I initially thought about this too and I checked the entire codebase for the implementations of get/set of `StructLike`. Pretty much all places don't check `pos` either in `get` or in `set`. The only exception is `PartitionData`, which only checks the `pos` in its `get` implementation [here](https://github.com/apache/iceberg/blob/fb02bd2d713c9bef8b8ccee334f6d65087c344b2/core/src/main/java/org/apache/iceberg/PartitionData.java#L133-L135). I'm ok adding a `pos` check in `set` but as I mentioned, we don't do this anywhere else across the codebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2412038049
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
there is really one valid reason: out of range field id (>1 M). if field id is within the 1M range, there is no ID overflow. Invalid field id (like <0) should really fail, as it is an internal error.
also nit on the error msg. maybe `Skip these fields when constructing content stats schema: {}`.
I am also wondering if we need to print the schema. The schema can be very large (>1M fields) if we really ran into the condition of skipping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2415194065
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
Review Comment:
thx for checking other code. I am good with leave it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409848261
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
Review Comment:
same comment as the other cases, I don't think we should throw in general here to avoid issues when reading/writing manifests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349149153
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void statsIdsOverflowForTableColumns() {
+ for (int i = 0; i < 100; i++) {
+ int id =
+ ThreadLocalRandom.current()
+ .nextInt(StatsUtil.METADATA_SPACE_FIELD_ID_START, StatsUtil.RESERVED_FIELD_IDS_START);
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = -1;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(id)).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ public void statsIdsForReservedColumns() {
+ int offset = 0;
+ for (int id = StatsUtil.RESERVED_FIELD_IDS_START; id < Integer.MAX_VALUE; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.METADATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset = offset + StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void contentStatsForSimpleSchema() {
+ Schema schema =
+ new Schema(
+ required(0, "i", Types.IntegerType.get()),
+ required(2, "f", Types.FloatType.get()),
+ required(4, "s", Types.StringType.get()),
+ required(6, "b", Types.BooleanType.get()),
+ required(250_000, "u", Types.UUIDType.get()));
+ Schema expectedStatsSchema =
+ new Schema(
+ optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ optional(10000, "0", fieldStatsFor(Types.IntegerType.get(), 10001)),
+ optional(10400, "2", fieldStatsFor(Types.FloatType.get(), 10401)),
+ optional(10800, "4", fieldStatsFor(Types.StringType.get(), 10801)),
+ optional(11200, "6", fieldStatsFor(Types.BooleanType.get(), 11201)),
+ optional(50010000, "250000", fieldStatsFor(Types.UUIDType.get(), 50010001)))));
+ Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
+ assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ }
+
+ @Test
+ public void contentStatsForComplexSchema() {
+ Schema schema =
+ new Schema(
+ required(0, "i", Types.IntegerType.get()),
+ required(2, "list", Types.ListType.ofOptional(3, Types.IntegerType.get())),
+ required(
+ 6,
+ "simple_struct",
+ Types.StructType.of(
+ optional(7, "int", Types.IntegerType.get()),
+ optional(8, "string", Types.StringType.get()))),
+ required(
+ 20,
+ "b",
+ Types.MapType.ofOptional(22, 24, Types.IntegerType.get(), Types.StringType.get())),
+ required(30, "variant", Types.VariantType.get()),
+ required(250_000, "u", Types.UUIDType.get()));
+ Schema expectedStatsSchema =
+ new Schema(
+ optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ optional(10000, "0", fieldStatsFor(Types.IntegerType.get(), 10001)),
+ optional(10600, "3", fieldStatsFor(Types.IntegerType.get(), 10601)),
+ optional(11400, "7", fieldStatsFor(Types.IntegerType.get(), 11401)),
+ optional(11600, "8", fieldStatsFor(Types.StringType.get(), 11601)),
+ optional(14400, "22", fieldStatsFor(Types.IntegerType.get(), 14401)),
+ optional(14800, "24", fieldStatsFor(Types.StringType.get(), 14801)),
+ optional(50010000, "250000", fieldStatsFor(Types.UUIDType.get(), 50010001)))));
+ Schema statsSchema = new Schema(StatsUtil.contentStatsFor(schema));
+ assertThat(statsSchema.asStruct()).isEqualTo(expectedStatsSchema.asStruct());
+ }
+
+ private Type fieldStatsFor(Type type, int id) {
+ int fieldId = id;
Review Comment:
here we're really only testing that the actual stats schema has the fields that we're expecting (we're not testing the actual stats themselves)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368788416
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ return javaClass.cast(fieldStats.get(pos));
+ }
+
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ BaseFieldStats stat = (BaseFieldStats) fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("column_size")) {
+ builder.columnSize((Long) record.getField("column_size"));
+ }
+ if (null != record.getField("value_count")) {
+ builder.valueCount((Long) record.getField("value_count"));
+ }
+ if (null != record.getField("nan_value_count")) {
+ builder.nanValueCount((Long) record.getField("nan_value_count"));
+ }
+ if (null != record.getField("null_value_count")) {
+ builder.nullValueCount((Long) record.getField("null_value_count"));
+ }
+ if (null != record.getField("lower_bound")) {
+ Object lowerBound = record.getField("lower_bound");
+ if (null != type) {
+ builder.lowerBound(type.typeId().javaClass().cast(lowerBound));
+ }
+ }
+ if (null != record.getField("upper_bound")) {
+ Object upperBound = record.getField("upper_bound");
+ if (null != type) {
+ builder.upperBound(type.typeId().javaClass().cast(upperBound));
+ }
+ }
+
+ BaseFieldStats newStat = builder.build();
+ fieldStats.set(pos, newStat);
+ } else {
+ fieldStats.set(pos, (FieldStats) value);
+ }
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", BaseContentStats.class.getSimpleName() + "[", "]")
Review Comment:
no reason, I've updated the code to use it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368292949
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
Shall we at least log a warn?
AFAIK, we didn't plan for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368544403
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
I commented about this in https://github.com/apache/iceberg/pull/13933#discussion_r2349170460 but do you think we have a strong use case for making `FieldStats` generic and use the parameterized type here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2480535546
##########
api/src/main/java/org/apache/iceberg/stats/FieldStatistic.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public enum FieldStatistic {
+ VALUE_COUNT(0, "value_count"),
+ NULL_VALUE_COUNT(1, "null_value_count"),
+ NAN_VALUE_COUNT(2, "nan_value_count"),
+ AVG_VALUE_SIZE(3, "avg_value_size"),
+ MAX_VALUE_SIZE(4, "max_value_size"),
+ LOWER_BOUND(5, "lower_bound"),
+ UPPER_BOUND(6, "upper_bound"),
+ IS_EXACT(7, "is_exact");
+
+ private final int offset;
+ private final String fieldName;
+
+ FieldStatistic(int offset, String fieldName) {
+ this.offset = offset;
+ this.fieldName = fieldName;
+ }
+
+ public int offset() {
+ return offset;
+ }
+
+ public String fieldName() {
+ return fieldName;
+ }
+
+ public static FieldStatistic fromOffset(int offset) {
+ switch (offset) {
+ case 0:
+ return VALUE_COUNT;
+ case 1:
+ return NULL_VALUE_COUNT;
+ case 2:
+ return NAN_VALUE_COUNT;
+ case 3:
+ return AVG_VALUE_SIZE;
+ case 4:
+ return MAX_VALUE_SIZE;
+ case 5:
+ return LOWER_BOUND;
+ case 6:
+ return UPPER_BOUND;
+ case 7:
+ return IS_EXACT;
+ default:
+ throw new IllegalArgumentException("Invalid statistic offset: " + offset);
+ }
+ }
+
+ public static Types.StructType fieldStatsFor(Type type, int fieldId) {
+ return Types.StructType.of(
+ optional(
+ fieldId + VALUE_COUNT.offset(),
+ VALUE_COUNT.fieldName(),
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ fieldId + NULL_VALUE_COUNT.offset(),
+ NULL_VALUE_COUNT.fieldName(),
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
Review Comment:
we can, but that implies that we have set the respective field statsStruct on each `FieldStats` instance so that we can properly fetch the right value whenever `get(int pos, Class<X> javaClass)` is called. This makes the impl more complicated and would technically require to set that statsStruct everywhere where a `FieldStats` instance is created.
This means that
```
BaseFieldStats<?> fieldStatsOne = BaseFieldStats.builder().fieldId(1).build();
BaseFieldStats<?> fieldStatsTwo = BaseFieldStats.builder().fieldId(2).build();
BaseContentStats stats =
BaseContentStats.builder()
.withTableSchema(...)
.withFieldStats(fieldStatsOne)
.withFieldStats(fieldStatsTwo)
.build();
```
would now become
```
BaseFieldStats<?> fieldStatsOne = BaseFieldStats.builder().fieldId(1).statsStruct(...).build();
BaseFieldStats<?> fieldStatsTwo = BaseFieldStats.builder().fieldId(2).statsStruct(...).build();
BaseContentStats stats =
BaseContentStats.builder()
.withTableSchema(...)
.withFieldStats(fieldStatsOne)
.withFieldStats(fieldStatsTwo)
.build();
```
This makes building a new `FieldStats` instance quite complicated. Another alternative would be to pass the `statsStruct` whenever the `ContentStats` instance is read. I did a quick POC and that would look something like this: https://github.com/apache/iceberg/commit/b676e1cdb22f485da1924fac61ca696bf884b74f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2412014169
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
Review Comment:
I assume we have a use case where the field stats was captured as a `GenericRecord`. wondering if we can move the code that convert from GenericRecord to BaseFieldStats in a util method. maybe a static method like `BaseFieldStats.from(GenericRecord)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2399582858
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
Review Comment:
does this interface needs to extends from `StructLike`?
I assume `content` means that this is also used for equality delete files (in addition to data files)? is `ContentFileStats` more informative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349099807
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
Review Comment:
oops yes, good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409353002
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
Review Comment:
we are simulating a StructLike with a list? does this class need to be `StructLike`?
I don't get the use case of get by position for StructLike here. `Class<T> javaClass` should always be a `FieldStats`.
I can probably see a reason for a getter by field id
```
public FieldStats<?> get(int fieldId)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409820508
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
Review Comment:
I don't think we should throw an exception because it's better to just not have any stats rather than failing to read/write the manifest just because we determined here that the field ID is < 0 (I don't think we prevent in the spec anywhere that field IDs can't be negative).
I forgot why I added `statsFieldId < 0` but you're right, this condition is already covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409854902
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
the reason why we skipped could have multiple reasons (invalid field id, ID overflow, ID out of range) so it's a little bit difficult to provide wording for that here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2310992195
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
Review Comment:
This one isn't as complicated as its inverse but I still would prefer to separate out the math into specific functions, that way we don't need the comment either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311055968
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void statsIdsOverflowForTableColumns() {
+ for (int i = 0; i < 100; i++) {
+ int id =
+ ThreadLocalRandom.current()
Review Comment:
No strong feeling here but why didn't you take this approach with testing non-overflow columns? I notice you do the full enumeration there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311052142
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
Review Comment:
I think this is meant to be NUM_STATS_PER_COLUMN, it just happens to work because they are both 200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2378605180
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
@pvary I revisited this and made this generic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2391984595
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
Review Comment:
Thanks for adding the check!
nit: add a test for this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2478525248
##########
api/src/main/java/org/apache/iceberg/stats/FieldStatistic.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public enum FieldStatistic {
+ VALUE_COUNT(0, "value_count"),
+ NULL_VALUE_COUNT(1, "null_value_count"),
+ NAN_VALUE_COUNT(2, "nan_value_count"),
+ AVG_VALUE_SIZE(3, "avg_value_size"),
+ MAX_VALUE_SIZE(4, "max_value_size"),
+ LOWER_BOUND(5, "lower_bound"),
+ UPPER_BOUND(6, "upper_bound"),
+ IS_EXACT(7, "is_exact");
+
+ private final int offset;
+ private final String fieldName;
+
+ FieldStatistic(int offset, String fieldName) {
+ this.offset = offset;
+ this.fieldName = fieldName;
+ }
+
+ public int offset() {
+ return offset;
+ }
+
+ public String fieldName() {
+ return fieldName;
+ }
+
+ public static FieldStatistic fromOffset(int offset) {
+ switch (offset) {
+ case 0:
+ return VALUE_COUNT;
+ case 1:
+ return NULL_VALUE_COUNT;
+ case 2:
+ return NAN_VALUE_COUNT;
+ case 3:
+ return AVG_VALUE_SIZE;
+ case 4:
+ return MAX_VALUE_SIZE;
+ case 5:
+ return LOWER_BOUND;
+ case 6:
+ return UPPER_BOUND;
+ case 7:
+ return IS_EXACT;
+ default:
+ throw new IllegalArgumentException("Invalid statistic offset: " + offset);
+ }
+ }
+
+ public static Types.StructType fieldStatsFor(Type type, int fieldId) {
+ return Types.StructType.of(
+ optional(
+ fieldId + VALUE_COUNT.offset(),
+ VALUE_COUNT.fieldName(),
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ fieldId + NULL_VALUE_COUNT.offset(),
+ NULL_VALUE_COUNT.fieldName(),
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
Review Comment:
Can't we exclude `NaN` if the type isn't floating point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368814079
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
I don't think we would want to log a warn here. What we could do however is signal at the place where the stats schema is later going to be defined that stats for field X are not available because the field ID is "invalid". That would happen in another PR though that builds up the stats schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368818370
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
also instead of checking < 0 it's actually better to check < 10_000 because stats field IDs will always be >= 10_000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2371971939
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
right now I'm doing it like this: https://github.com/apache/iceberg/pull/13694/files#diff-1e7f52f095238038cd87a2b3b434815feeb11d95c9134d8c1ecd53da41904fceR311-R330
I'd probably still hold off with the parameterized type for now since we can always add it in one of the follow-up PRs if we see that there's actual use of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2372262185
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
Why would you not log in case we have unexpected values for the stat ids?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2411931989
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
> Instead it's better to indicate that there are no stats available for field ID X
this is the other direction of mapping from stats field id to a data field id. if stats field id is not in the expected range, shouldn't it be an error?
Can we have a statsFieldId not matching the condition?
```
statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2415199239
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
agree. It is consistent with our principle that writer should be strictly correct, but readers should be more liberal on accepting non-compliant files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2411931989
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
> Instead it's better to indicate that there are no stats available for field ID X
this is the other direction of mapping from stats field id to a data field id. if stats field id is not in the expected range, shouldn't it be an error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311006012
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
Review Comment:
Paranoia again, but maybe we should specifically name all these offsets rather than relying on the order in the code here.
ie
COLUMN_SIZE_OFFSET = 0
NAN_VALUE_OFFSET = 1
...
I assume we are going to need those somewhere anyway for extraction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349084191
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
Review Comment:
makes sense, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349140851
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
Review Comment:
makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368343982
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ return javaClass.cast(fieldStats.get(pos));
+ }
+
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ BaseFieldStats stat = (BaseFieldStats) fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("column_size")) {
+ builder.columnSize((Long) record.getField("column_size"));
+ }
+ if (null != record.getField("value_count")) {
Review Comment:
nit: shall we add newlines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2371981240
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
sorry, my previous answer was wrong/misleading because I forgot that I moved all of the stats schema code. Basically if we would want to log a warn then this would happen in https://github.com/apache/iceberg/pull/13933/files#diff-5467d1f9b780966e1d3dd2b3c32dc745a3988038046d8f3c1e90d739223a0aaaR184 and this is the place that uses the `-1` but I'm still undecided whether it's a good idea to actually do that warn or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2412026635
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ FieldStats<?> stat = fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("value_count")) {
Review Comment:
so the `GenericRecord` can only partially update some stats? is that the use case we are trying to support with `GenericRecord` set value? if yes, earlier comment for static `BaseFieldStats.from(GenericRecord)` method doesn't make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2411994440
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
Review Comment:
thx for explaining. maybe add comments to briefly explain the reason of returning null for out of range position.
`< 0` is probably an invalid condition that can fail with Preconditions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413743360
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
purely from the Java impl we should never hit this condition, because we control how stats field IDs are generated. This is mostly to prevent us from cases where e.g. other Iceberg clients accidentally used wrong stats field IDs. If that ever happens and we throw here, the manifest will be unreadable, which is what we want to avoid. Thus we need to be lenient on enforcing such things.
We took a similar approach when implementing views, where we are lenient when reading potentially invalid metadata that doesn't fully adhere to the Spec just to avoid having unreadable metadata. That being said, we should do the same here, does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413779883
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "Field stats must be a struct type: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type = null;
+ if (null != structType.field("lower_bound")) {
+ type = structType.field("lower_bound").type();
+ } else if (null != structType.field("upper_bound")) {
+ type = structType.field("upper_bound").type();
+ }
+
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Wrong class, expected %s but was %s for object: %s",
+ javaClass.getName(), value.getClass().getName(), value));
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Override
+ public <T> void set(int pos, T value) {
+ if (value instanceof GenericRecord) {
+ GenericRecord record = (GenericRecord) value;
+ FieldStats<?> stat = fieldStats.get(pos);
+ BaseFieldStats.Builder builder = BaseFieldStats.buildFrom(stat);
+ Type type = stat.type();
+ if (null != record.getField("value_count")) {
Review Comment:
Yes, `GenericRecord` can also just hold one or two fields and this part is called by Avro code when reading a manifest. Depending on which stats are available, the `GenericRecord` will just hold values for those stats. I'm not sure it's worth putting this into a util method, since there's really no other use case for re-using this anywhere else other than here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368257465
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
Should this be defined by generic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2482161611
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
Review Comment:
We should add comments since this is confusing and related to the metadata file field reserved space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #13933:
URL: https://github.com/apache/iceberg/pull/13933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2425317497
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
+
+ /** A list of all the {@link FieldStats} */
+ List<FieldStats<?>> fieldStats();
Review Comment:
yep I agree and this already exists in the POC PR: https://github.com/apache/iceberg/pull/13694/files#diff-00cf59714ba243d384d0d986d9a1613a7028a4579fc71faa50014d5ca191482aR27
I didn't see any need to introduce this here because we don't have any actual code that would call this atm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409762039
##########
api/src/main/java/org/apache/iceberg/stats/ContentStats.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+
+public interface ContentStats extends StructLike {
Review Comment:
yes this needs to extend `StructLike` and the reason I didn't put `File` in the name is because we want to use this not only for files but also for other things, like partition stats and such
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409398335
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
Review Comment:
if `pos` is out of range, should we fail here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2412038049
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
there is really one valid reason: out of range field id (>1 M). if field id is within the 1M range, there is no ID overflow. Invalid field id (like <0) should really fail, as it is an internal error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409372939
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats<T> implements FieldStats<T>, StructLike, Serializable {
Review Comment:
nit: `FieldStats` interface already extends from `StructLike`. redundant here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2457669672
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats<T> implements FieldStats<T>, Serializable {
+ private final int fieldId;
+ private final Type type;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final Integer avgValueSize;
+ private final Integer maxValueSize;
+ private final T lowerBound;
+ private final T upperBound;
+
+ private BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ Integer avgValueSize,
+ Integer maxValueSize,
+ T lowerBound,
+ T upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.avgValueSize = avgValueSize;
+ this.maxValueSize = maxValueSize;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Type type() {
+ return type;
+ }
+
+ @Override
+ public Long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public Long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public Long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Integer avgValueSize() {
+ return avgValueSize;
+ }
+
+ @Override
+ public Integer maxValueSize() {
+ return maxValueSize;
+ }
+
+ @Override
+ public T lowerBound() {
+ return lowerBound;
+ }
+
+ @Override
+ public T upperBound() {
+ return upperBound;
+ }
+
+ @Override
+ public int size() {
+ return 6;
+ }
+
+ @Override
+ public <X> X get(int pos, Class<X> javaClass) {
+ switch (pos) {
Review Comment:
I feel like a `ContentStatistic` enum would make this a little clearner as well since you could use `switch (ContentStatistic.forOffset(pos())) . . . `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349106368
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void statsIdsOverflowForTableColumns() {
+ for (int i = 0; i < 100; i++) {
+ int id =
+ ThreadLocalRandom.current()
Review Comment:
typically I try to avoid randomness in tests because it makes debugging more difficult. However, the table ID range here is large and the test takes very long to complete, thus I opted to use 100 random IDs from that table ID range
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2371942554
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
+ int fieldId();
+
+ Type type();
+
+ Long columnSize();
+
+ Long valueCount();
+
+ Long nullValueCount();
+
+ Long nanValueCount();
+
+ Object lowerBound();
Review Comment:
> I commented about this in [#13933 (comment)](https://github.com/apache/iceberg/pull/13933#discussion_r2349170460)
Sorry, I have missed that.
> but do you think we have a strong use case for making `FieldStats` generic and use the parameterized type here?
Maybe when we are applying the filters in the ManifestEvalVisitor:
```
@Override
public <T> Boolean eq(BoundReference<T> ref, Literal<T> lit) {
int pos = Accessors.toPosition(ref.accessor());
PartitionFieldSummary fieldStats = stats.get(pos);
if (fieldStats.lowerBound() == null) {
return ROWS_CANNOT_MATCH; // values are all null and literal cannot contain null
}
T lower = Conversions.fromByteBuffer(ref.type(), fieldStats.lowerBound());
int cmp = lit.comparator().compare(lower, lit.value());
if (cmp > 0) {
return ROWS_CANNOT_MATCH;
}
T upper = Conversions.fromByteBuffer(ref.type(), fieldStats.upperBound());
cmp = lit.comparator().compare(upper, lit.value());
if (cmp < 0) {
return ROWS_CANNOT_MATCH;
}
return ROWS_MIGHT_MATCH;
}
```
This could be like this:
```
@Override
public <T> Boolean eq(BoundReference<T> ref, Literal<T> lit) {
int pos = Accessors.toPosition(ref.accessor());
FieldStats<T> fieldStats = (FieldStats<T>) stats.get(pos);
if (fieldStats.lowerBound() == null) {
return ROWS_CANNOT_MATCH; // values are all null and literal cannot contain null
}
if (lit.comparator().compare(fieldStats.lowerBound(), lit.value()) > 0) {
return ROWS_CANNOT_MATCH;
}
if (lit.comparator().compare(fieldStats.upperBound(), lit.value()) < 0) {
return ROWS_CANNOT_MATCH;
}
return ROWS_MIGHT_MATCH;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368553300
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
technically this should never happen, but you never know whether a different client impl provided a wrong stats field ID that we cannot correctly map back to the table field ID space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2310977868
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
Review Comment:
nit on the name, statsIdForField?
I may be being paranoid here but I feel like we are putting a lot of math in here and at least to me it's obscuring things a bit.
Maybe instead we could do
```
statsIdForField (fieldId) {
Precondition (fieldID >= 0)
return (fieldId >= RESEREVED_FIELD_ID_START) ?
statsIdForReservedField(fieldId) :
statsIdForDataField(fieldId)
}
statsIdForDataField(fieldId) {
statsId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) id;
//overflow conditions specific to data field
}
statsIdForReservedField(fieldId) {
id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldID)
statId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * id
// Overflow check for ReservedFields
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2310679083
##########
api/src/main/java/org/apache/iceberg/stats/FieldStats.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.types.Type;
+
+public interface FieldStats extends StructLike {
Review Comment:
Maybe we could parameterize this? Then we could have a strongly typed FieldStats object. I'm thinking mostly about lowerBound() and upperBound()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2311057179
##########
api/src/test/java/org/apache/iceberg/stats/TestStatsUtil.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestStatsUtil {
+
+ @Test
+ public void statsIdsForTableColumns() {
+ int offset = 0;
+ // 10_000 + 200 * 7_084_950 = 1_417_000_000, which is the starting range for reserved columns
+ int max = (StatsUtil.METADATA_SPACE_FIELD_ID_START - StatsUtil.DATA_SPACE_FIELD_ID_START) / 200;
+ for (int id = 0; id < max; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.DATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset += StatsUtil.RESERVED_FIELD_IDS;
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(id);
+ }
+ }
+
+ @Test
+ public void statsIdsOverflowForTableColumns() {
+ for (int i = 0; i < 100; i++) {
+ int id =
+ ThreadLocalRandom.current()
+ .nextInt(StatsUtil.METADATA_SPACE_FIELD_ID_START, StatsUtil.RESERVED_FIELD_IDS_START);
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = -1;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(id)).as("at pos %s", id).isEqualTo(expected);
+ assertThat(StatsUtil.fieldIdFor(statsFieldId)).as("at pos %s", id).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ public void statsIdsForReservedColumns() {
+ int offset = 0;
+ for (int id = StatsUtil.RESERVED_FIELD_IDS_START; id < Integer.MAX_VALUE; id++) {
+ int statsFieldId = StatsUtil.statsFieldIdFor(id);
+ int expected = StatsUtil.METADATA_SPACE_FIELD_ID_START + offset;
+ assertThat(statsFieldId).as("at pos %s", id).isEqualTo(expected);
+ offset = offset + StatsUtil.RESERVED_FIELD_IDS;
Review Comment:
I think this is also supposed to be NUM_STATS_PER_COLUMN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2482157003
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int NUM_STATS_PER_COLUMN = 200;
Review Comment:
`NUM_RESERVED_STATS_PER_COLUMN`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2464858393
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
+ if (fieldId >= 0) {
+ Types.StructType structType = contentStatsFor(field.type(), fieldId + 1);
+ return optional(fieldId, Integer.toString(field.fieldId()), structType);
+ } else {
+ skippedFieldIds.add(field.fieldId());
+ }
+
+ return null;
+ }
+
+ @Override
+ public Types.NestedField variant(Types.VariantType variant) {
+ return null;
Review Comment:
For the lower/upper bound of a Variant we will store the actual `VariantObject`. The stats evaluator can then look up the respective variant value given the normalized path. This is also how it works today already, except that the `VariantObject` is stored as a `ByteBuffer`. See also [this commit](https://github.com/apache/iceberg/pull/13694/commits/30ac70e40bcdc233e7bd2ece03afeb1b424f9bf4) that shows how this is going to be implemented later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2382662199
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats<T> implements FieldStats<T>, StructLike, Serializable {
+ private final transient int fieldId;
+ private final transient Type type;
+ private final Long columnSize;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final T lowerBound;
+ private final T upperBound;
+
+ BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long columnSize,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ T lowerBound,
+ T upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.columnSize = columnSize;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Type type() {
+ return type;
+ }
+
+ @Override
+ public Long columnSize() {
+ return columnSize;
+ }
+
+ @Override
+ public Long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public Long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public Long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public T lowerBound() {
+ return lowerBound;
+ }
+
+ @Override
+ public T upperBound() {
+ return upperBound;
+ }
+
+ @Override
+ public int size() {
+ return 6;
+ }
+
+ @Override
+ public <X> X get(int pos, Class<X> javaClass) {
+ switch (pos) {
Review Comment:
We might be able to check the javaClass here also to match the type
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats<T> implements FieldStats<T>, StructLike, Serializable {
+ private final transient int fieldId;
+ private final transient Type type;
+ private final Long columnSize;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final T lowerBound;
+ private final T upperBound;
+
+ BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long columnSize,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ T lowerBound,
+ T upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.columnSize = columnSize;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
Review Comment:
Question: Do we want to check that the Type, and `T` are matching?
Something like:
```
Preconditions.checkArgument(lowerBound == null ||
lowerBound.getClass().isInstance(type.typeId().javaClass()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2397680560
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
+ if (value == null || javaClass.isInstance(value)) {
+ return javaClass.cast(value);
+ }
+
+ throw new IllegalArgumentException(
Review Comment:
I was planning to add this later in the follow-up PR but fair enough and I think it makes sense to add this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2457689552
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn("Could not create stats schema for field ids: {}", visitor.skippedFieldIds);
+ }
+
+ return result;
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ return Types.StructType.of(
+ optional(
+ id + VALUE_COUNT_OFFSET,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(
+ id + NULL_VALUE_COUNT_OFFSET,
+ "null_value_count",
+ Types.LongType.get(),
+ "Total null value count"),
+ optional(
+ id + NAN_VALUE_COUNT_OFFSET,
+ "nan_value_count",
+ Types.LongType.get(),
+ "Total NaN value count"),
+ optional(
+ id + AVG_VALUE_SIZE_OFFSET,
+ "avg_value_size",
+ Types.IntegerType.get(),
+ "Avg value size of variable-length types (String, Binary)"),
+ optional(
+ id + MAX_VALUE_SIZE_OFFSET,
+ "max_value_size",
+ Types.IntegerType.get(),
+ "Max value size of variable-length types (String, Binary)"),
+ optional(id + LOWER_BOUND_OFFSET, "lower_bound", type, "Lower bound"),
+ optional(id + UPPER_BOUND_OFFSET, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+ private final Set<Integer> skippedFieldIds = Sets.newLinkedHashSet();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
+ "content_stats",
+ Types.StructType.of(
+ statsFields.stream()
+ .filter(Objects::nonNull)
+ .sorted(Comparator.comparing(Types.NestedField::fieldId))
+ .collect(Collectors.toList())));
+ }
+
+ @Override
+ public Types.NestedField list(Types.ListType list, Types.NestedField elementResult) {
+ list.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField map(
+ Types.MapType map, Types.NestedField keyResult, Types.NestedField valueResult) {
+ map.fields()
+ .forEach(
+ field -> {
+ Types.NestedField result = field(field, null);
+ if (null != result) {
+ statsFields.add(result);
+ }
+ });
+ return null;
+ }
+
+ @Override
+ public Types.NestedField struct(Types.StructType struct, List<Types.NestedField> fields) {
+ statsFields.addAll(fields);
+ return null;
+ }
+
+ @Override
+ public Types.NestedField field(Types.NestedField field, Types.NestedField fieldResult) {
+ if (field.type().isNestedType() || field.type().isVariantType()) {
+ return null;
+ }
+
+ int fieldId = StatsUtil.statsFieldIdForField(field.fieldId());
+ if (fieldId >= 0) {
+ Types.StructType structType = contentStatsFor(field.type(), fieldId + 1);
+ return optional(fieldId, Integer.toString(field.fieldId()), structType);
+ } else {
+ skippedFieldIds.add(field.fieldId());
+ }
+
+ return null;
+ }
+
+ @Override
+ public Types.NestedField variant(Types.VariantType variant) {
+ return null;
Review Comment:
Have we not defined the representation for variant? Would we just carry forward the binary representation we have in V3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2349092539
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) finalId;
+ }
+
+ public static int fieldIdFor(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ int finalId;
+ if (statsFieldId < METADATA_SPACE_FIELD_ID_START) {
+ finalId = (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ } else {
+ // this is a reserved field ID, which uses a different calculation
+ finalId =
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN;
+ }
+
+ return Math.max(-1, finalId);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ return TypeUtil.visit(schema, new ContentStatsSchemaVisitor());
+ }
+
+ private static Types.StructType contentStatsFor(Type type, int id) {
+ int fieldId = id;
+
+ return Types.StructType.of(
+ optional(fieldId++, "column_size", Types.LongType.get(), "Total size on disk"),
+ optional(
+ fieldId++,
+ "value_count",
+ Types.LongType.get(),
+ "Total value count, including null and NaN"),
+ optional(fieldId++, "nan_value_count", Types.LongType.get(), "Total NaN value count"),
+ optional(fieldId++, "null_value_count", Types.LongType.get(), "Total null value count"),
+ optional(fieldId++, "lower_bound", type, "Lower bound"),
+ optional(fieldId, "upper_bound", type, "Upper bound"));
+ }
+
+ private static class ContentStatsSchemaVisitor extends TypeUtil.SchemaVisitor<Types.NestedField> {
+ private final List<Types.NestedField> statsFields = Lists.newArrayList();
+
+ @Override
+ public Types.NestedField schema(Schema schema, Types.NestedField structResult) {
+ return optional(
+ 146,
Review Comment:
yes that constant will eventually be defined once we have the final v4 structure in place. In my other PR that contains a temp. v4 structure it's defined in https://github.com/apache/iceberg/pull/13694/files#diff-e4322d37add71d01166c1b34721bf2a1e9366c1b901ebdec944dd90bc19d0806R124-R126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2310988679
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ private static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 1_417_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdFor(int fieldId) {
+ int idSpaceStart = DATA_SPACE_FIELD_ID_START;
+ int id = fieldId;
+ if (fieldId >= RESERVED_FIELD_IDS_START) {
+ // this is a reserved field ID, which uses a different calculation
+ idSpaceStart = METADATA_SPACE_FIELD_ID_START;
+ id = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+ }
+
+ long finalId = idSpaceStart + NUM_STATS_PER_COLUMN * (long) id;
+ if (finalId < 0
+ || finalId > RESERVED_FIELD_IDS_START
+ || (finalId >= METADATA_SPACE_FIELD_ID_START
+ && idSpaceStart != METADATA_SPACE_FIELD_ID_START)) {
+ // ID overflows
+ return -1;
Review Comment:
Why don't we just break here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2407926629
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
while we can get ID overflow for the direction of mapping field id to stats field id, we probably should throw an `IllegalArgumentException` for the other direction if `statsFieldId` doesn't conform to the expected validations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413757058
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
> Invalid field id (like <0) should really fail, as it is an internal error.
I believe we need to be lenient with invalid field IDs as we don't want to fail manifest reading/writing. See also my other [comment](https://github.com/apache/iceberg/pull/13933#discussion_r2413743360) on this topic.
> I am also wondering if we need to print the schema. The schema can be very large (>1M fields) if we really ran into the condition of skipping
Yes you're right, schema could become quite large, so maybe it's better to remove it from the msg. I'd prefer to have `Could not create stats schema for field ids: {}` as that better aligns with other error messages across the codebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2413757058
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
+ static final int NAN_VALUE_COUNT_OFFSET = 2;
+ static final int AVG_VALUE_SIZE_OFFSET = 3;
+ static final int MAX_VALUE_SIZE_OFFSET = 4;
+ static final int LOWER_BOUND_OFFSET = 5;
+ static final int UPPER_BOUND_OFFSET = 6;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+ static final int MAX_DATA_STATS_FIELD_ID = 200_010_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId < 0 || fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < DATA_SPACE_FIELD_ID_START || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
+ }
+
+ return statsFieldId < METADATA_SPACE_FIELD_ID_START
+ ? fieldIdForStatsFieldFromDataField(statsFieldId)
+ : fieldIdForStatsFieldFromReservedField(statsFieldId);
+ }
+
+ private static int fieldIdForStatsFieldFromDataField(int statsFieldId) {
+ return Math.max(-1, (statsFieldId - DATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ private static int fieldIdForStatsFieldFromReservedField(int statsFieldId) {
+ return Math.max(
+ -1,
+ statsFieldId
+ - RESERVED_FIELD_IDS
+ + (Integer.MAX_VALUE - statsFieldId)
+ + (statsFieldId - METADATA_SPACE_FIELD_ID_START) / NUM_STATS_PER_COLUMN);
+ }
+
+ public static Types.NestedField contentStatsFor(Schema schema) {
+ ContentStatsSchemaVisitor visitor = new ContentStatsSchemaVisitor();
+ Types.NestedField result = TypeUtil.visit(schema, visitor);
+ if (!visitor.skippedFieldIds.isEmpty()) {
+ LOG.warn(
+ "Could not create stats schema for field ids {} of schema: {}",
Review Comment:
> Invalid field id (like <0) should really fail, as it is an internal error.
I believe we need to be lenient with invalid field IDs as we don't want to fail manifest reading/writing. See also my other [comment](https://github.com/apache/iceberg/pull/13933#discussion_r2413743360) on this topic.
> I am also wondering if we need to print the schema. The schema can be very large (>1M fields) if we really ran into the condition of skipping
Yes you're right, schema could become quite large, so maybe it's better to remove it from the msg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2409783794
##########
core/src/main/java/org/apache/iceberg/stats/BaseContentStats.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class BaseContentStats implements ContentStats, StructLike, Serializable {
+
+ private final List<FieldStats<?>> fieldStats;
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public BaseContentStats(Types.StructType projection) {
+ this.fieldStats = Lists.newArrayListWithCapacity(projection.fields().size());
+ for (int i = 0; i < projection.fields().size(); i++) {
+ Types.NestedField field = projection.fields().get(i);
+ Preconditions.checkArgument(
+ field.type().isStructType(), "ColumnStats must contain structs: %s", field.type());
+ Types.StructType structType = field.type().asStructType();
+ Type type =
+ null != structType.field("lower_bound")
+ ? structType.field("lower_bound").type()
+ : null != structType.field("upper_bound")
+ ? structType.field("upper_bound").type()
+ : null;
+ fieldStats.add(
+ BaseFieldStats.builder()
+ .fieldId(StatsUtil.fieldIdForStatsField(field.fieldId()))
+ .type(type)
+ .build());
+ }
+ }
+
+ private BaseContentStats(List<FieldStats<?>> fieldStats) {
+ this.fieldStats = Lists.newArrayList(fieldStats);
+ }
+
+ @Override
+ public List<FieldStats<?>> fieldStats() {
+ return fieldStats;
+ }
+
+ @Override
+ public int size() {
+ return fieldStats.size();
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ if (pos < 0 || pos > fieldStats().size() - 1) {
+ return null;
+ }
+
+ FieldStats<?> value = fieldStats.get(pos);
Review Comment:
yes this needs to be a `StructLike` and this part has to do with how things are read by Avro. Things will get clearer when looking at the PR that contains the full POC work and verifies that everything works with Avro, which is https://github.com/apache/iceberg/pull/13694 but we can't currently use that one because the way we map the v4 structure is still WIP and we haven't defined yet where the `content_stats` object will be hooked into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2421623532
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StatsUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(StatsUtil.class);
+ static final int VALUE_COUNT_OFFSET = 0;
+ static final int NULL_VALUE_COUNT_OFFSET = 1;
Review Comment:
I feel like we might want the FieldStats defined as an Enum in the FieldStats class. This would allow us to specifically project the fields that we care about based on a reference to the specific stat. As an int, it's harder to call that out in a projection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2368295656
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ public static int fieldIdForStatsField(int statsFieldId) {
+ if (statsFieldId < 0 || statsFieldId % NUM_STATS_PER_COLUMN != 0) {
+ return -1;
Review Comment:
When does this happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2376415551
##########
core/src/main/java/org/apache/iceberg/stats/BaseFieldStats.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import java.io.Serializable;
+import java.util.Objects;
+import java.util.StringJoiner;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class BaseFieldStats implements FieldStats, StructLike, Serializable {
+ private final transient int fieldId;
+ private final transient Type type;
+ private final Long columnSize;
+ private final Long valueCount;
+ private final Long nullValueCount;
+ private final Long nanValueCount;
+ private final Object lowerBound;
+ private final Object upperBound;
+
+ BaseFieldStats(
+ int fieldId,
+ Type type,
+ Long columnSize,
+ Long valueCount,
+ Long nullValueCount,
+ Long nanValueCount,
+ Object lowerBound,
+ Object upperBound) {
+ this.fieldId = fieldId;
+ this.type = type;
+ this.columnSize = columnSize;
+ this.valueCount = valueCount;
+ this.nullValueCount = nullValueCount;
+ this.nanValueCount = nanValueCount;
+ this.lowerBound = lowerBound;
+ this.upperBound = upperBound;
+ }
+
+ @Override
+ public int fieldId() {
+ return fieldId;
+ }
+
+ @Override
+ public Type type() {
+ return type;
+ }
+
+ @Override
+ public Long columnSize() {
+ return columnSize;
+ }
+
+ @Override
+ public Long valueCount() {
+ return valueCount;
+ }
+
+ @Override
+ public Long nullValueCount() {
+ return nullValueCount;
+ }
+
+ @Override
+ public Long nanValueCount() {
+ return nanValueCount;
+ }
+
+ @Override
+ public Object lowerBound() {
+ return lowerBound;
+ }
+
+ @Override
+ public Object upperBound() {
+ return upperBound;
+ }
+
+ @Override
+ public int size() {
+ return 6;
+ }
+
+ @Override
+ public Object get(int pos, Class javaClass) {
+ switch (pos) {
+ case StatsUtil.COLUMN_SIZE_OFFSET:
+ return javaClass.cast(columnSize);
+ case StatsUtil.VALUE_COUNT_OFFSET:
+ return javaClass.cast(valueCount);
+ case StatsUtil.NULL_VALUE_COUNT_OFFSET:
+ return javaClass.cast(nullValueCount);
+ case StatsUtil.NAN_VALUE_COUNT_OFFSET:
+ return javaClass.cast(nanValueCount);
+ case StatsUtil.LOWER_BOUND_OFFSET:
+ return javaClass.cast(lowerBound);
+ case StatsUtil.UPPER_BOUND_OFFSET:
+ return javaClass.cast(upperBound);
+ default:
+ throw new UnsupportedOperationException("Unknown field ordinal: " + pos);
+ }
+ }
+
+ @Override
+ public void set(int pos, Object value) {
+ throw new UnsupportedOperationException("set() not supported");
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", BaseFieldStats.class.getSimpleName() + "[", "]")
Review Comment:
oops, yes absolutely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2374696668
##########
api/src/main/java/org/apache/iceberg/stats/StatsUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public class StatsUtil {
+ static final int COLUMN_SIZE_OFFSET = 0;
+ static final int VALUE_COUNT_OFFSET = 1;
+ static final int NULL_VALUE_COUNT_OFFSET = 2;
+ static final int NAN_VALUE_COUNT_OFFSET = 3;
+ static final int LOWER_BOUND_OFFSET = 4;
+ static final int UPPER_BOUND_OFFSET = 5;
+ static final int NUM_STATS_PER_COLUMN = 200;
+ static final int RESERVED_FIELD_IDS = 200;
+ static final int DATA_SPACE_FIELD_ID_START = 10_000;
+ static final int METADATA_SPACE_FIELD_ID_START = 2_147_000_000;
+ static final int RESERVED_FIELD_IDS_START = Integer.MAX_VALUE - RESERVED_FIELD_IDS;
+ // support stats for only up to this amount of data field IDs
+ static final int MAX_DATA_FIELD_ID = 1_000_000;
+
+ private StatsUtil() {}
+
+ public static int statsFieldIdForField(int fieldId) {
+ return fieldId >= RESERVED_FIELD_IDS_START
+ ? statsFieldIdForReservedField(fieldId)
+ : statsFieldIdForDataField(fieldId);
+ }
+
+ private static int statsFieldIdForDataField(int fieldId) {
+ long statsFieldId = DATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) fieldId;
+ if (fieldId > MAX_DATA_FIELD_ID || statsFieldId < 0) {
+ return -1;
+ }
+
+ return (int) statsFieldId;
+ }
+
+ private static int statsFieldIdForReservedField(int fieldId) {
+ int offset = RESERVED_FIELD_IDS - (Integer.MAX_VALUE - fieldId);
+
+ long statsFieldId = METADATA_SPACE_FIELD_ID_START + NUM_STATS_PER_COLUMN * (long) offset;
+ if (statsFieldId < 0 || statsFieldId > RESERVED_FIELD_IDS_START) {
+ // ID overflows
Review Comment:
I think the only concern I had with the warn is if a user had a large schema with lots and lots of "invalid" (e.g. > 1_000_000) field IDs but practically speaking I don't think this will ever be a concern in a real use case. I've added a WARN stating that the stats schema for those field IDs couldn't be created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2478522501
##########
api/src/main/java/org/apache/iceberg/stats/FieldStatistic.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public enum FieldStatistic {
+ VALUE_COUNT(0, "value_count"),
+ NULL_VALUE_COUNT(1, "null_value_count"),
+ NAN_VALUE_COUNT(2, "nan_value_count"),
+ AVG_VALUE_SIZE(3, "avg_value_size"),
+ MAX_VALUE_SIZE(4, "max_value_size"),
+ LOWER_BOUND(5, "lower_bound"),
+ UPPER_BOUND(6, "upper_bound"),
+ IS_EXACT(7, "is_exact");
Review Comment:
Should we use `exact_bounds` or `tight_bounds?. I think the `exact` only applies to the bounds. The other fields should always be exact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Introduce classes for content stats [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13933:
URL: https://github.com/apache/iceberg/pull/13933#discussion_r2478590183
##########
api/src/main/java/org/apache/iceberg/stats/FieldStatistic.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.stats;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public enum FieldStatistic {
+ VALUE_COUNT(0, "value_count"),
+ NULL_VALUE_COUNT(1, "null_value_count"),
+ NAN_VALUE_COUNT(2, "nan_value_count"),
+ AVG_VALUE_SIZE(3, "avg_value_size"),
+ MAX_VALUE_SIZE(4, "max_value_size"),
+ LOWER_BOUND(5, "lower_bound"),
+ UPPER_BOUND(6, "upper_bound"),
+ IS_EXACT(7, "is_exact");
Review Comment:
`exact_bounds` makes sense to me, I've renamed things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org