You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ajantha-bhat (via GitHub)" <gi...@apache.org> on 2024/04/18 15:48:40 UTC

[PR] Data: Add a util to read write partition stats [iceberg]

ajantha-bhat opened a new pull request, #10176:
URL: https://github.com/apache/iceberg/pull/10176

   Based on the conclusion from https://github.com/apache/iceberg/pull/9437, we are going with the local implementation with multi threads instead of distributive algorithm. 
   
   `iceberg-core` module cannot write stats as it doesn't depend on parquet code. 
   So, `iceberg-data` module is introducing an API to write it and core module just registers it to the table. 
   
   Syntax look like this.
   ```
   GeneratePartitionStats computePartitionStats = new GeneratePartitionStats(testTable);
   PartitionStatisticsFile result = computePartitionStats.generate();
   testTable.updatePartitionStatistics().setPartitionStatistics(result).commit();
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571024420


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {

Review Comment:
   `iceberg-core` module cannot write stats as it doesn't depend on parquet code.
   So, `iceberg-data` module is introducing an API to write it and core module just registers it to the table.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679714714


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+
+  GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    // TODO: Support an option to accept branches and tag info and lookup corresponding snapshot.
+    Snapshot currentSnapshot = table.currentSnapshot();
+    if (currentSnapshot == null) {
+      return null;
+    }
+
+    // Map of partitionData, partition-stats-entry per partitionData
+    Map<Record, Record> partitionEntryMap = Maps.newConcurrentMap();
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    List<ManifestFile> manifestFiles = currentSnapshot.allManifests(table.io());
+    Tasks.foreach(manifestFiles)
+        .suppressFailureWhenFinished()
+        .executeWith(ThreadPools.getWorkerPool())
+        .onFailure(
+            (file, thrown) ->
+                LOG.warn(

Review Comment:
   We aren't going to fail if the manifest is broken?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679576861


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   Why do we need to convert this? PartitionData is already an IndexedRecord,StructLike



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698398597


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);

Review Comment:
   Can change it to use `Enum.name()` instead of ordinal with `getField()` and `setField()`.  But `getField` returns `Object` and needs explicit casting. Hence, Russell also suggested to use get() with classType to avoid casting in some comment long back. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1724417472


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;

Review Comment:
   Updated now.
   
   2 million data file entires. 
   
   100 partitions * 20 data files per partitions = 2k data files per manifest. 
   10K manifests. 
   
   It took around 10 seconds to write, commit and read the partition stats file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711350427


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;

Review Comment:
   branch or tags are finally mapped to snapshot. So, Once the query engine resolves the branch/tag to a particular snapshot, it can check/compute the stats for that snapshot and track it from table metadata. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1701879499


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   @lirui-apache: You are right about needing extra conversion for few data types. I have added the test case for all the data type as partition now in `TestPartitionStatsWriterUtil#testAllDatatypePartition` and also improved `IdentityPartitionConverters`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1777461986


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   @lirui-apache: You were right that we shouldn't combine the stats for different spec. Just that we didn't had a strong usecase. 
   
   Anton mentioned that 
   > If we combine delete stats across specs, they will not be accurate. A delete file from one spec never applies to a data file in another spec.
   
   So, now we are not combining it. 
   
   more details on https://github.com/apache/iceberg/pull/11146 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1705461680


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    // Sorting the records based on partition as per spec.
+    Map<Record, Record> partitionEntryMap =
+        new ConcurrentSkipListMap<>(Comparators.forType(partitionType));

Review Comment:
   Added the JMH benchmarks, for 30K manifests (1 add, 2 update operation per partition value * 10k partitions), it took only 0.5 seconds with concurrent hashmap. This includes reading 30k manifests in multi threaded way and computing the concurrent, sorting the map and writing the parquet partition stats and committing to the table. 
   
   I wanted to try 100K partitions but the problem is generating 30k manifests and committing them to table took around 30 minutes. So, it takes lot of setup time if I increase the manifests. 
   
   Since 0.5 seconds is very small. Do we still need to validate concurrent hashmap vs Threadlocal hashmap performance? 
   
   
   ```
   Benchmark                                             Mode  Cnt  Score   Error  Units
   PartitionStatsGeneratorBenchmark.writePartitionStats    ss    5  0.556 ± 0.343   s/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711350427


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;

Review Comment:
   branch or tags are finally mapped to snapshot. So, Once the query engine resolves the branch/tag to a particular snapshot, it can check/compute the stats for that snapshot and track it from table metadata. 
   
   So, I don't think this branch info has to be stored in the stats file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2258726699

   I should finally have some time today. @ajantha-bhat, is PR #9437 still valid as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571033881


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   the idea of this field is to apply the equality or position delete filters and give the effective results. Need to add a logic to compute it in the follow up. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1687524246


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   I see. I assumed it will sort the data as we are passing to writer. Looks like it just adds sort order to metadata. 
   
   During Spec design phase, we decided that it is good to sort the partition stats before writing to a file so that it can help for faster lookup and better compression. But seems like we need engine to sort it. Just the Java API of parquet writers we have doesn't have the option to sort it?
   
   So, we should update back the spec that these fields cannot be sorted in a separate PR. WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697730639


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),

Review Comment:
   Does selecting live entries affect the computation of `last_updated_at`? Are we able to catch operations that modify partitions by removing data files?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1691044481


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   Maybe there're other data types we need to convert. E.g. the generic writer for `Date` expects a `LocalDate`, while the internal class for `Date` is an integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1642810642


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   yes. please refer to the last point of https://github.com/apache/iceberg/pull/9437#issuecomment-2064341275
   
   Also, it is similar to how the partitions metadata table implemented currently.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721979347


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;

Review Comment:
   Why is it called iteration counter? Just a little confusing since we are in a benchmark class which already defines iterations as the number of times we run the test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722723867


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests

Review Comment:
   Included the ITERATION_COUNTER, I will reword as 10000 manifest per iteration. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1723873509


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {

Review Comment:
   Could we doc this and Writer Util? I'm mostly interested in the separation of these two classes and what should their responsibilities be?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679558702


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns

Review Comment:
   I would remove these comments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2316864310

   > Add PartitionStatsWriter that would wrap PartitionStats as Record
   
   This abstraction isn't working out. 
   Because `PartitionStats` need partition as `PartitionData` and `Record` need partition as `Record`. 
   Converting them on wrapping and unwrapping is becoming very messy because storage is same and there are multiple paths to store it. 
   
   I am thinking to introduce `PartitionStatsRecord` that extends `Record` and store it in `Core` similar to `GenericRecord` and make the interfaces based on that in `core` and `data` instead of `PartitionStats`. 
   
   Thoughts? 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711348696


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =

Review Comment:
   For writer generic writer was available. For reader it is not available. Hence, it is a TODO. 
   I have mentioned the same at https://github.com/apache/iceberg/pull/10176#discussion_r1571025684
   
   The idea is to have a generic reader in a separate PR and enhance this feature to use it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1642551527


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   Does this mean we allow merging records with different spec IDs? Suppose we start with `Spec1: {Field#1}`. And we write a partition `Filed#1 = 1`. Then we change spec to `Spec2: {Field#1, Field#2}`, and write another partition `Field#1 = 1 / Field#2 = null`. Will we only have one entry in the partition stats file with `Partition_Data: {Field#1 = 1 / Field#2 = null}, Spec_ID: 2`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721885650


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.

Review Comment:
   Nit here and other places in the doc, the doc string is probably clearer as 
   "Generates the Partition Stats Files Schema based on a given a partition type" 
   
   Should we also note this should be called with the unified spec? Or is it ok to use with a non-unified spec?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721988655


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;
+
+  private Table table;
+
+  private PartitionStatisticsFile result;
+
+  @Setup
+  public void setupBenchmark() throws IOException {
+    baseDir =
+        Paths.get(new File(System.getProperty("java.io.tmpdir")).getAbsolutePath()).toString();
+    table = TestTables.create(new File(baseDir), "foo", SCHEMA, SPEC, SortOrder.unsorted(), 2);
+
+    for (int interations = 0; interations < ITERATION_COUNTER; interations++) {
+      for (int partitionOrdinal = 0; partitionOrdinal < PARTITION_COUNT; partitionOrdinal++) {
+        StructLike partition = TestHelpers.Row.of(partitionOrdinal);
+        AppendFiles appendFiles = table.newAppend();
+        for (int fileOrdinal = 0; fileOrdinal < DATA_FILES_PER_PARTITION_COUNT; fileOrdinal++) {
+          DataFile dataFile = FileGenerationUtil.generateDataFile(table, partition);
+          appendFiles.appendFile(dataFile);
+        }
+
+        appendFiles.commit();
+      }
+    }
+  }
+
+  @TearDown
+  public void tearDownBenchmark() throws IOException {
+    // validate row count
+    try (CloseableIterable<Record> recordIterator =

Review Comment:
   I don't think this is an appropriate place to assert that the benchmark ran correctly, one issue here is a failed run will not clean up the temp folder. 
   
   Is there a reason to not place this in the benchmark itself?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1687515919


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   True. But we decided we need to have this optional field during spec design. 
   
   Plan is to have a follow up PR with a flag that enables computing this. By default it won't be computed as it is an expensive operation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1687545473


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   Because the parquet writer expects an Iceberg `Record` and `IndexedRecord` is not an Iceberg `Record`
   
   ```
   java.lang.ClassCastException: class org.apache.iceberg.PartitionData cannot be cast to class org.apache.iceberg.data.Record (org.apache.iceberg.PartitionData and org.apache.iceberg.data.Record are in unnamed module of loader 'app')
   	at org.apache.iceberg.data.parquet.GenericParquetWriter$RecordWriter.get(GenericParquetWriter.java:41)
   	at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:580)
   	at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:581)
   	at org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:135)
   	at org.apache.iceberg.io.DataWriter.write(DataWriter.java:71)
   	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
   	at org.apache.iceberg.data.PartitionStatsWriterUtil.writePartitionStatsFile(PartitionStatsWriterUtil.java:82)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1692578140


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679827491


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {

Review Comment:
   minor request, we should test the null case for all the optional fields 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722738419


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+  private Types.StructType partitionType;
+  private Map<Record, Record> partitionEntryMap;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.

Review Comment:
   Previously the declaration was here. Missed to move the comment. I will update it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697297242


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");

Review Comment:
   Question: Are we saying it is illegal to compute partition stats for unpartitioned tables given that most of the information is actually available in the snapshot summary? In other words, there is no value in partition stats for unpartitioned tables?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2147007461

   @RussellSpitzer, @szehon-ho: It seems Anton is on holiday for 2-3 weeks. Is it possible for any of you to support for review? 
   
   cc: @jbonofre   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1687927207


##########
data/src/test/java/org/apache/iceberg/data/TestComputePartitionStats.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.assertj.core.groups.Tuple;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestComputePartitionStats {

Review Comment:
   Added all these. Will add some more by tomorrow for equality and position delete stats and validation for snapshot ids etc. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698471050


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);

Review Comment:
   No. Empty branches will have a dummy snapshot without manifest files. 
   
   https://github.com/apache/iceberg/blob/76dba8fe83c4496318cc34436e610cf50f43054d/core/src/main/java/org/apache/iceberg/SnapshotManager.java#L81
   
   So, I am not throwing exception for that case. 
   Added test and It creates an empty stats file for the dummy snapshot since snapshot exist. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2335041347

   Let me take a look hopefully over weekend or on Monday.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat closed pull request #10176: Data: Add a util to read write partition stats
URL: https://github.com/apache/iceberg/pull/10176


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679689988


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag
+    return record;
+  }
+
+  private static PartitionData coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    PartitionData data = new PartitionData(partitionType);
+    for (int i = 0; i < partitionType.fields().size(); i++) {
+      Object val = partition.get(i, partitionType.fields().get(i).type().typeId().javaClass());
+      if (val != null) {
+        data.set(i, val);
+      }
+    }
+    return data;
+  }
+
+  private static List<String> scanColumns(ManifestContent content) {

Review Comment:
   Should this be in BaseScan? This may be a fine location as well just wondering if it belongs here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1678859302


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   I don't think this really does the sorting. Seems it only marks the generated file with the sort ID?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679547549


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,

Review Comment:
   nit: Just called "partition" in the Spec



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571032953


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {

Review Comment:
   if the snapshot is expired, last updated time and last updated snapshot will be null. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1644433728


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   let us go with the example, 
   partitioned with `col1` and inserted 5 rows for that partition `col1 = x`. 
   now the spec has evolved with partition as `col1/col2` and inserted 3 rows for `col1 = x` and `col2 = y`
   and inserted 7 rows for `col1 = x` and `col2 = null`
   
   So, when the query is run with `col1 = x` and `col2 = null`, it will fetch 12 (5+7) rows. So, the partition stats will also have stats from 12 rows for `col1 = x` and `col2 = null` (unified tuple). So, I think it is fine. 
   
   Can you elaborate why do you think it is a problem to have unified tuple? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698068822


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   True. Maybe we didn't think well about this field during spec design. We can drop a todo (add a note instead) and make this field deprecated if required later on. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698508423


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  /**
+   * Converts the given {@link PartitionData} into a {@link Record} based on the specified partition
+   * schema.
+   *
+   * @param partitionSchema the schema defining the structure of the partition data.
+   * @param partitionData the data to be converted into a Record.
+   * @return a Record that represents the partition data as per the given schema.
+   */
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(Column.PARTITION.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);

Review Comment:
   Null, as mentioned in https://github.com/apache/iceberg/pull/10176#discussion_r1698066417, 
   
   also added the tests. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1727881929


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   I understand why we mark `LAST_UPDATED_SNAPSHOT_ID` and `LAST_UPDATED_AT ` as optional. My question was about counts. In my view, null means unknown and 0 means absent. If I read a partition stats file and see null, it won't be informative.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   I understand why we mark `LAST_UPDATED_SNAPSHOT_ID` and `LAST_UPDATED_AT` as optional. My question was about counts. In my view, null means unknown and 0 means absent. If I read a partition stats file and see null, it won't be informative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721980729


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;

Review Comment:
   Seems like this would be very unrealistic for benchmarking? Shouldn't we have multiple data files in each manifest (also multiple partitions in the same manifest?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1705560828


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =

Review Comment:
   The reader only supports Parquet at the moment. Why we make the writer support different formats?



##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    Map<Record, Record> partitionEntryMap = Maps.newConcurrentMap();
+
+    Schema dataSchema = PartitionStatsUtil.schema(partitionType);
+    List<ManifestFile> manifestFiles = currentSnapshot.allManifests(table.io());
+    Tasks.foreach(manifestFiles)
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .onFailure(
+            (file, thrown) ->
+                LOG.warn(
+                    "Failed to compute the partition stats for the manifest file: {}",
+                    file.path(),
+                    thrown))
+        .run(
+            manifest -> {
+              try (CloseableIterable<Record> entries =
+                  PartitionStatsUtil.fromManifest(table, manifest, dataSchema)) {
+                entries.forEach(
+                    entry ->
+                        partitionEntryMap.compute(

Review Comment:
   nit: I think it's more natural to use `merge` here than `compute`



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   Not sure whether the conversion should happen here, or just before the data is fed to the writers. Does it make any difference whether we want the comparators/map to operate on internal or generic types?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679698799


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+
+  GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    // TODO: Support an option to accept branches and tag info and lookup corresponding snapshot.

Review Comment:
   Why not just have this method accept a "Snapshot" object instead of just using the "current"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679815600


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("id").build();
+    Table testTable =
+        TestTables.create(
+            temp.newFolder("test_partition_stats"),
+            "test_partition_stats",
+            SCHEMA,
+            spec,
+            SortOrder.unsorted(),
+            2);
+
+    Types.StructType partitionSchema = Partitioning.partitionType(testTable);
+    Schema dataSchema = PartitionStatsUtil.schema(partitionSchema);
+
+    ImmutableList.Builder<Record> partitionListBuilder = ImmutableList.builder();
+
+    long seed = System.currentTimeMillis();
+    LOG.info("Seed used for random generator is {}", seed);
+    Random random = new Random(seed);
+
+    for (int i = 0; i < 42; i++) {
+      PartitionData partitionData =
+          new PartitionData(
+              dataSchema.findField(Column.PARTITION_DATA.name()).type().asStructType());
+      partitionData.set(0, random.nextLong());
+
+      Record record = GenericRecord.create(dataSchema);
+      record.set(
+          Column.PARTITION_DATA.ordinal(),
+          PartitionStatsUtil.partitionDataToRecord(partitionSchema, partitionData));
+      record.set(Column.SPEC_ID.ordinal(), random.nextInt(10));
+      record.set(Column.DATA_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.DATA_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), 1024L * random.nextInt(20));
+      record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.TOTAL_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), random.nextLong());
+
+      partitionListBuilder.add(record);
+    }
+    List<Record> records = partitionListBuilder.build();
+
+    OutputFile outputFile = PartitionStatsWriterUtil.newPartitionStatsFile(testTable, 42L);
+    PartitionStatsWriterUtil.writePartitionStatsFile(testTable, records.iterator(), outputFile);
+
+    Assertions.assertThat(Paths.get(outputFile.location())).exists();
+
+    List<Record> rows;
+    try (CloseableIterable<Record> recordIterator =
+        PartitionStatsWriterUtil.readPartitionStatsFile(
+            dataSchema, Files.localInput(outputFile.location()))) {
+      rows = Lists.newArrayList(recordIterator);
+    }
+
+    Assertions.assertThat(rows).hasSize(records.size());

Review Comment:
   This whole block can use the assertIterableEquals
   ```java
    Assertions.assertIterableEquals();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679810070


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("id").build();
+    Table testTable =
+        TestTables.create(
+            temp.newFolder("test_partition_stats"),
+            "test_partition_stats",
+            SCHEMA,
+            spec,
+            SortOrder.unsorted(),
+            2);
+
+    Types.StructType partitionSchema = Partitioning.partitionType(testTable);
+    Schema dataSchema = PartitionStatsUtil.schema(partitionSchema);
+
+    ImmutableList.Builder<Record> partitionListBuilder = ImmutableList.builder();
+
+    long seed = System.currentTimeMillis();
+    LOG.info("Seed used for random generator is {}", seed);
+    Random random = new Random(seed);
+
+    for (int i = 0; i < 42; i++) {
+      PartitionData partitionData =
+          new PartitionData(
+              dataSchema.findField(Column.PARTITION_DATA.name()).type().asStructType());
+      partitionData.set(0, random.nextLong());
+
+      Record record = GenericRecord.create(dataSchema);
+      record.set(
+          Column.PARTITION_DATA.ordinal(),
+          PartitionStatsUtil.partitionDataToRecord(partitionSchema, partitionData));
+      record.set(Column.SPEC_ID.ordinal(), random.nextInt(10));
+      record.set(Column.DATA_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.DATA_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), 1024L * random.nextInt(20));
+      record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.TOTAL_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), random.nextLong());
+
+      partitionListBuilder.add(record);
+    }
+    List<Record> records = partitionListBuilder.build();
+
+    OutputFile outputFile = PartitionStatsWriterUtil.newPartitionStatsFile(testTable, 42L);
+    PartitionStatsWriterUtil.writePartitionStatsFile(testTable, records.iterator(), outputFile);
+
+    Assertions.assertThat(Paths.get(outputFile.location())).exists();
+
+    List<Record> rows;

Review Comment:
   nit: writtenRecords



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679809693


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("id").build();
+    Table testTable =
+        TestTables.create(
+            temp.newFolder("test_partition_stats"),
+            "test_partition_stats",
+            SCHEMA,
+            spec,
+            SortOrder.unsorted(),
+            2);
+
+    Types.StructType partitionSchema = Partitioning.partitionType(testTable);
+    Schema dataSchema = PartitionStatsUtil.schema(partitionSchema);
+
+    ImmutableList.Builder<Record> partitionListBuilder = ImmutableList.builder();
+
+    long seed = System.currentTimeMillis();
+    LOG.info("Seed used for random generator is {}", seed);
+    Random random = new Random(seed);
+
+    for (int i = 0; i < 42; i++) {
+      PartitionData partitionData =
+          new PartitionData(
+              dataSchema.findField(Column.PARTITION_DATA.name()).type().asStructType());
+      partitionData.set(0, random.nextLong());
+
+      Record record = GenericRecord.create(dataSchema);
+      record.set(
+          Column.PARTITION_DATA.ordinal(),
+          PartitionStatsUtil.partitionDataToRecord(partitionSchema, partitionData));
+      record.set(Column.SPEC_ID.ordinal(), random.nextInt(10));
+      record.set(Column.DATA_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.DATA_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), 1024L * random.nextInt(20));
+      record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), random.nextInt());
+      record.set(Column.TOTAL_RECORD_COUNT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), random.nextLong());
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), random.nextLong());
+
+      partitionListBuilder.add(record);
+    }
+    List<Record> records = partitionListBuilder.build();

Review Comment:
   nit: expectedRecords



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679833286


##########
data/src/test/java/org/apache/iceberg/data/TestComputePartitionStats.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.assertj.core.groups.Tuple;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestComputePartitionStats {

Review Comment:
   I think we should have some tests
   
   1. Tests where the snapshot is empty
   2. Tests where some of stats are null
   3. Tests where the partition spec has changed for some of the files



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679568700


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {

Review Comment:
   Method name here is pretty vague, also anything public should have a javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1729159956


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2343906548

   > I just learned that there is an ongoing effort to support writes with the internal Iceberg object model, which will nicely solve this problem for us (given that it is already in progress, it shouldn't delay this work).
   
   Could you please point me to that work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698339571


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),

Review Comment:
   I wanted to avoid duplicate stats, for example when compacted or for rewrite manifests, old manifests will be marked as `DELETED`. Hence, keeping liveEntries will help skipping those manifests. 
   
   Also, if the manifests are marked as `DELETED`, it won't be considered for table scan. So, since these stats are used for query planning, I think it is safe to skip these. Thoughts? 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2261017015

   Thanks @RussellSpitzer, @aokolnychyi and @lirui-apache for the review feedbacks. 
   I have also added strong test validations today.
   
   I have addressed or replied to each comment. 
   
   As per me, pending things are
   a) https://github.com/apache/iceberg/pull/10176#discussion_r1691044481, need to check about record converter and more unit tests for record converter for each data type. 
   b) JMH benchmark for concurrent map logic to try out performance optimizations. 
   I will check these tomorrow. 
   
   Let me know if you can do another pass. Lets not lose momentum on this again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571037034


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   sorting as per the unified partition tuple order as per the spec.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1727994442


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null

Review Comment:
   If we assume expired snapshots are older (which is mostly true unless someone expires a snapshot in the middle of the chain), then this logic seems to work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1727901172


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   It makes sense to keep fields optional but I would expect implementations to write 0 for the position delete record count if a partition has no deletes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2311678409

   > Add a wrapper class to represent partition stats in core.
   
   I didn't use the wrapper class initially and directly used record because for each entry having two objects (wrapper and Record) might increase memory footprint. But looking at benchmarks, it may not be a deal breaker and it will look clean as you suggested. 
   
   I will update the code as you suggested. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679828614


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("id").build();
+    Table testTable =
+        TestTables.create(
+            temp.newFolder("test_partition_stats"),
+            "test_partition_stats",
+            SCHEMA,
+            spec,
+            SortOrder.unsorted(),
+            2);
+
+    Types.StructType partitionSchema = Partitioning.partitionType(testTable);
+    Schema dataSchema = PartitionStatsUtil.schema(partitionSchema);
+
+    ImmutableList.Builder<Record> partitionListBuilder = ImmutableList.builder();
+
+    long seed = System.currentTimeMillis();

Review Comment:
   In general, unless we are doing an explicit property test we tend to make sure the tests are consistent. So I would just freeze the seed as a constant in the file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1704749308


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;

Review Comment:
   We would be storing one partition stats file per snapshot + branch combination? Also looks like we don't have branch information in spec?



##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(

Review Comment:
   I think the properties are not associated to any snapshot. In which case we wouldn't know the format with which it was written for a specific snapshot?
   



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.IdentityPartitionConverters;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),

Review Comment:
   Here are we referring to the spec id corresponding to the snapshot for which stats are collected? (We store the unified partition spec, but also store spec_id. The spec referred to will not be same as the one stored in stats files, is that intended?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722735848


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;

Review Comment:
   It takes lot of time to generate manifests (for 10k manifests it took around 20 minutes). 
   I will optimize it for a balanced number and user can increase the counters if they want to see benchmarks against more manifests with less data files or less manifests with more data file per manifests. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722827850


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;
+
+  private Table table;
+
+  private PartitionStatisticsFile result;
+
+  @Setup
+  public void setupBenchmark() throws IOException {
+    baseDir =
+        Paths.get(new File(System.getProperty("java.io.tmpdir")).getAbsolutePath()).toString();
+    table = TestTables.create(new File(baseDir), "foo", SCHEMA, SPEC, SortOrder.unsorted(), 2);
+
+    for (int interations = 0; interations < ITERATION_COUNTER; interations++) {
+      for (int partitionOrdinal = 0; partitionOrdinal < PARTITION_COUNT; partitionOrdinal++) {
+        StructLike partition = TestHelpers.Row.of(partitionOrdinal);
+        AppendFiles appendFiles = table.newAppend();
+        for (int fileOrdinal = 0; fileOrdinal < DATA_FILES_PER_PARTITION_COUNT; fileOrdinal++) {
+          DataFile dataFile = FileGenerationUtil.generateDataFile(table, partition);
+          appendFiles.appendFile(dataFile);
+        }
+
+        appendFiles.commit();
+      }
+    }
+  }
+
+  @TearDown
+  public void tearDownBenchmark() throws IOException {
+    // validate row count
+    try (CloseableIterable<Record> recordIterator =

Review Comment:
   Wanted to benchmark just write performance without any read code. But you are right. I have to move it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722931839


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStats(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    Record partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);

Review Comment:
   Currently `recordSchema` was computed by guarded by `PartitionStatsUtil.schema()` and passed to this method. Hence, it was guarded. 
   
   But as it is a public API, better to add the check here also. I will update it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721954085


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStats(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    Record partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);

Review Comment:
   Does this work with an unpartitioned  spec? or are we guarding against it elsewhere?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721978424


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests

Review Comment:
   Unclear where the 3 is coming from here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1687976550


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   How about using a sorted map to store the stats entries? I suppose we can get a comparator for the partition data using the `Comparators.forType` util, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697760344


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    // Sorting the records based on partition as per spec.
+    Map<Record, Record> partitionEntryMap =
+        new ConcurrentSkipListMap<>(Comparators.forType(partitionType));

Review Comment:
   I am worried about the overhead of maintaining the map sorted on writes (there will be a lot of writes). I wonder whether using `ConcurrentHashMap` and sorting once at the end will be more efficient. Can we add a JMH benchmark to verify this?
   
   If we notice thread-safe collections cause significant overhead, we can switch to aggregating manifests concurrently and doing one more pass for the final aggregation. Then we will be able to avoid concurrency problems without overhead. We will only be able to tell if thread-safe collections cause significant overhead after having a good benchmark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697992679


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    // Sorting the records based on partition as per spec.
+    Map<Record, Record> partitionEntryMap =
+        new ConcurrentSkipListMap<>(Comparators.forType(partitionType));

Review Comment:
   @lirui-apache, You are right. I pushed a new test now which validates each and every stats. I observed flakiness because of  `ConcurrentSkipListMap#compute`. 
   
   I will sort after populating the map as suggested by you and Anton. JMH, I will add in a follow up PR. I think we can incrementally improve this logic based on JMH benchmarks later on. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571030511


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   observe optional and required fields, as per spec. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2098057287

   ping @aokolnychyi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698339815


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");

Review Comment:
   updated. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1644281052


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   Hmm.. but how can such a stats entry be useful? From a user's perspective, there's no way to tell if an entry is merged. Suppose the entry says we have 10 records for partition `{Field#1 = 1 / Field#2 = null}`. This doesn't mean there're 10 records satisfying filter `Field#1 = 1 and Field#2 is null`, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1644433728


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   lets us go with the example, 
   partitioned with `col1` and inserted 5 rows for that partition `col1 = x`. 
   now the spec has evolved with partition as `col1/col2` and inserted 3 rows for `col1 = x` and `col2 = y`
   and inserted 7 rows for `col1 = x` and `col2 = null`
   
   So, when the query is run with `col1 = x` and `col2 = null`, it will fetch 12 (5+7) rows. So, the partition stats will also have stats from 12 rows for `col1 = x` and `col2 = null` (unified tuple). So, I think it is fine. 
   
   Can you elaborate why do you think it is a problem to have unified tuple? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1692578855


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   I will take a look. Is there a util class to handle this already? Because isn't it a common problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1692587758


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   I believe `IdentityPartitionConverters` can be used to do the conversion. And perhaps we can have test cases for date/time partition columns?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697282703


##########
core/src/main/java/org/apache/iceberg/BaseScan.java:
##########
@@ -289,4 +289,22 @@ private static Schema lazyColumnProjection(TableScanContext context, Schema sche
   public ThisT metricsReporter(MetricsReporter reporter) {
     return newRefinedScan(table, schema, context.reportWith(reporter));
   }
+
+  /**
+   * Retrieves a list of column names based on the type of manifest content provided.
+   *
+   * @param content the manifest content type, which specifies whether the columns are for data or

Review Comment:
   Minor: What about shortening the description here to stay on one line? 
   Like simply `the manifest content type to scan`?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.

Review Comment:
   Minor: `a Schema object` -> `a schema`?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");

Review Comment:
   Question: Is there a particular reason why we are not using `Preconditions` here?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   Question: We made these counts nullable cause we think not all implementations will populate these values? We will still write 0 if needed in the current implementation, though? Except the total record count, which may be expensive to compute.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");

Review Comment:
   Question: Are we saying it is illegal to compute partition stats for unpartitioned tables given that most of the information is actually available in the snapshot summary?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);

Review Comment:
   It feels a bit fragile to use the enum ordinals. Let me think.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null

Review Comment:
   Hm, can we actually compute `last_updated_at` if at least one side has null? I thought such computations will require knowing this value for all records.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.

Review Comment:
   Minor: `a Schema object` -> `a schema`?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  /**
+   * Converts the given {@link PartitionData} into a {@link Record} based on the specified partition
+   * schema.
+   *
+   * @param partitionSchema the schema defining the structure of the partition data.
+   * @param partitionData the data to be converted into a Record.
+   * @return a Record that represents the partition data as per the given schema.
+   */
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(Column.PARTITION.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);

Review Comment:
   If there are no deletes, what kind of values will we have as delete counts? Null or 0?



##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {

Review Comment:
   We usually name classes as nouns. What about something like `PartitionStatsGenerator`?



##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);

Review Comment:
   Can we have empty branches (without snapshots)?



##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    // Sorting the records based on partition as per spec.
+    Map<Record, Record> partitionEntryMap =
+        new ConcurrentSkipListMap<>(Comparators.forType(partitionType));

Review Comment:
   I am worried about the overhead of maintaining the map sorted on writes (there will be a lot of writes). I wonder whether using `ConcurrentHashMap` and sorting once at the end will be more efficient. Can we add a JMH benchmark to verify this?
   
   If we notice thread-safe collections cause significant overhead, we can switch to aggregating manifests concurrently and doing one more pass for the final aggregation. Then we will be able to avoid concurrency problems without overhead.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),

Review Comment:
   Does selecting live entries affect the computation of `last_updated_at`?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   I am not sure we would want to extend this path to compute deletes. We may also reconsider how position deletes are handled in V3, so it may be possible to compute this without scanning the data.
   
   I'd probably drop this TODO for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714691005


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(

Review Comment:
   I didn't get it fully. 
   
   Do you mean, we should also store the output file format type in the `PartitionStatisticsFile` for the reader to be known? Currently based on the file extension, we can know which reader to be used. Hence, it is not explicitly stored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1727966586


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),

Review Comment:
   The problem is that the computed `last_updated_at` may no longer be accurate (we won't catch operations that modified partitions by removing data files). We can project the entry status and only take deleted entires into account when computing the last updated timestamp. Does it worth the extra complexity? We'll have to think.
   
   > compacted or for rewrite manifests, old manifests will be marked as DELETED
   
   Don't we actually remove such manifests from the manifest list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721966029


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStats(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    Record partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(Column.PARTITION.ordinal(), partitionData);
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // Note: Not computing the `TOTAL_RECORD_COUNT` for now as it needs scanning the data.
+    return record;
+  }
+
+  private static Record coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    GenericRecord genericRecord = GenericRecord.create(partitionType);
+    for (int index = 0; index < partitionType.fields().size(); index++) {
+      genericRecord.set(
+          index,
+          partition.get(index, partitionType.fields().get(index).type().typeId().javaClass()));
+    }
+
+    return genericRecord;
+  }
+
+  private static void checkAndIncrementLong(Record toUpdate, Record fromRecord, Column column) {
+    if (fromRecord.get(column.ordinal()) != null) {

Review Comment:
   Personal prefrence from me here but I find it clearer (albeit longer) to do
   ```java
   if (from !=null && to !=null) {
      to = from + to
   } else if (from != null) {
      to = from;
   }
   ```
   
   Just to decrease the nesting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721957929


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStats(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    Record partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(Column.PARTITION.ordinal(), partitionData);
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // Note: Not computing the `TOTAL_RECORD_COUNT` for now as it needs scanning the data.

Review Comment:
   Couldn't we do this as long as there are no Position or Equality Deletes? Since we are doing an aggregation we could do something like adding up the total record count but then setting it to null if we ever see a position or equality delete? 
   
   Could be a follow up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722038556


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {
+  private static final Logger LOG = LoggerFactory.getLogger(PartitionStatsGenerator.class);
+
+  private final Table table;
+  private String branch;
+  private Types.StructType partitionType;
+  private Map<Record, Record> partitionEntryMap;
+
+  public PartitionStatsGenerator(Table table) {
+    this.table = table;
+  }
+
+  public PartitionStatsGenerator(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.

Review Comment:
   Comment (if needed) should probably be on the declaration 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2292897105

   Thanks a lot for the review @lirui-apache, 
   
   I hope we can get this in after a review from @RussellSpitzer and @aokolnychyi.   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679562033


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));

Review Comment:
   Why did we need to do a copy if we are going to create a new object and discard the manifest entry here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679549740


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,

Review Comment:
   called: total_data_file_size_in_bytes in the spec



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679666190


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag
+    return record;
+  }
+
+  private static PartitionData coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    PartitionData data = new PartitionData(partitionType);
+    for (int i = 0; i < partitionType.fields().size(); i++) {
+      Object val = partition.get(i, partitionType.fields().get(i).type().typeId().javaClass());
+      if (val != null) {
+        data.set(i, val);
+      }
+    }
+    return data;
+  }
+
+  private static List<String> scanColumns(ManifestContent content) {
+    switch (content) {
+      case DATA:
+        return BaseScan.SCAN_COLUMNS;
+      case DELETES:
+        return BaseScan.DELETE_SCAN_COLUMNS;
+      default:
+        throw new UnsupportedOperationException("Cannot read unknown manifest type: " + content);
+    }
+  }
+
+  private static void incrementLong(Record toUpdate, Record fromRecord, Column column) {
+    toUpdate.set(
+        column.ordinal(),
+        (long) toUpdate.get(column.ordinal()) + (long) fromRecord.get(column.ordinal()));
+  }
+
+  private static void checkAndIncrementLong(Record toUpdate, Record fromRecord, Column column) {
+    if (toUpdate.get(column.ordinal()) != null && fromRecord.get(column.ordinal()) != null) {

Review Comment:
   I also think we don't gain a lot by splitting out the check here, do we have a usecase for incrementLong without checking?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2339212241

   Okay, I played around with this a bit over the weekend. I suggest we split this PR into two parts: `PartitionStatsUtil` that computes and sorts the stats (1) and the conversion and writing of the stats into a file (2). The reason for the split is that we may need to reconsider the writing a bit. The current problem is the need to translate to generic record for writing into Parquet. I just learned that there is an ongoing effort to support writes with the internal Iceberg object model, which will nicely solve this problem for us (given that it is already in progress, it shouldn't delay this work).
   
   While we look into the writing part, it would be nice to get `PartitionStatsUtil` in. For that, I would suggest making some tweaks to `PartitionStats` and `PartitionStatsUtil`:
   
   - Use `StructLike` instead of `Record`.
   - Don't create a temporary `PartitionStats` from a manifest entry to simply merge it with others. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2377498681

   Splitted this into two PRs as suggested.
   
   compute:
   https://github.com/apache/iceberg/pull/11146
   
   Write:
   https://github.com/apache/iceberg/pull/11216
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1727881929


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   I understand why we mark `LAST_UPDATED_SNAPSHOT_ID` and `LAST_UPDATED_AT` as optional. My question was about counts. In my view, null means unknown and 0 means absent. If I read a partition stats file and see null as the number of position deletes, it won't be informative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1728009901


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsGeneratorUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // Create 10k manifests
+  private static final int MANIFEST_COUNTER = 10000;
+
+  // each manifest with 100 partition values
+  private static final int PARTITION_PER_MANIFEST = 100;
+
+  // 20 data files per partition, which results in 2k data files per manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 20;
+
+  private Table table;
+
+  @Setup
+  public void setupBenchmark() {
+    baseDir =
+        Paths.get(new File(System.getProperty("java.io.tmpdir")).getAbsolutePath()).toString();
+    table = TestTables.create(new File(baseDir), "foo", SCHEMA, SPEC, SortOrder.unsorted(), 2);
+
+    IntStream.range(0, MANIFEST_COUNTER)

Review Comment:
   It would be nice to parallelize the preparation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2102416207

   cc: @RussellSpitzer, @szehon-ho : Can you please take a look? I need reviews and I want to take it forward.
   previous discussions can be found at https://github.com/apache/iceberg/pull/9437 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679617189


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   That would be tricky, not sure we can actually do it without actually scanning rows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679728924


##########
data/src/test/java/org/apache/iceberg/data/TestComputePartitionStats.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.assertj.core.groups.Tuple;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestComputePartitionStats {

Review Comment:
   Needs tests covering the null behaviors I mentioned above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679665205


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag
+    return record;
+  }
+
+  private static PartitionData coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    PartitionData data = new PartitionData(partitionType);
+    for (int i = 0; i < partitionType.fields().size(); i++) {
+      Object val = partition.get(i, partitionType.fields().get(i).type().typeId().javaClass());
+      if (val != null) {
+        data.set(i, val);
+      }
+    }
+    return data;
+  }
+
+  private static List<String> scanColumns(ManifestContent content) {
+    switch (content) {
+      case DATA:
+        return BaseScan.SCAN_COLUMNS;
+      case DELETES:
+        return BaseScan.DELETE_SCAN_COLUMNS;
+      default:
+        throw new UnsupportedOperationException("Cannot read unknown manifest type: " + content);
+    }
+  }
+
+  private static void incrementLong(Record toUpdate, Record fromRecord, Column column) {
+    toUpdate.set(
+        column.ordinal(),
+        (long) toUpdate.get(column.ordinal()) + (long) fromRecord.get(column.ordinal()));
+  }
+
+  private static void checkAndIncrementLong(Record toUpdate, Record fromRecord, Column column) {
+    if (toUpdate.get(column.ordinal()) != null && fromRecord.get(column.ordinal()) != null) {

Review Comment:
   If toUpdate is null but fromRecord is not null shouldn't we set toUpdate to fromRecord?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1682214561


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag
+    return record;
+  }
+
+  private static PartitionData coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    PartitionData data = new PartitionData(partitionType);
+    for (int i = 0; i < partitionType.fields().size(); i++) {
+      Object val = partition.get(i, partitionType.fields().get(i).type().typeId().javaClass());
+      if (val != null) {
+        data.set(i, val);
+      }
+    }
+    return data;
+  }
+
+  private static List<String> scanColumns(ManifestContent content) {
+    switch (content) {
+      case DATA:
+        return BaseScan.SCAN_COLUMNS;
+      case DELETES:
+        return BaseScan.DELETE_SCAN_COLUMNS;
+      default:
+        throw new UnsupportedOperationException("Cannot read unknown manifest type: " + content);
+    }
+  }
+
+  private static void incrementLong(Record toUpdate, Record fromRecord, Column column) {
+    toUpdate.set(
+        column.ordinal(),
+        (long) toUpdate.get(column.ordinal()) + (long) fromRecord.get(column.ordinal()));
+  }
+
+  private static void checkAndIncrementLong(Record toUpdate, Record fromRecord, Column column) {
+    if (toUpdate.get(column.ordinal()) != null && fromRecord.get(column.ordinal()) != null) {

Review Comment:
   > I also think we don't gain a lot by splitting out the check here, do we have a usecase for incrementLong without checking?
   
   There are some optional fields and some required fields, I wanted to check null only for optional fields. Hence I had two separate methods to avoid null check for required fields.  But it may not matter much. I will simplify. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1650446414


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   @ajantha-bhat Thanks for clarifying! Personally I still think this feature can be used to improve user experience for investigating partitions. Maybe we just need to explain to users about the ambiguity for partitions with null values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1647514744


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(

Review Comment:
   > So maybe my question should be whether the partition stats are only intended for query optimizers, and not suitable for our use case
   
   It was designed mainly thinking query planners in mind. The spec was approved for unified tuple. The reason is query planner has to scan both the folders in this case [(c1 = 1), (c1 = 1 and c2 = null)], So the unified tuple include stats from both the spec. keeping stats per individual spec might slowdown the planning time to compute the unified stats. 
   
   the stats like file count and record count is mainly used for CBO and last modified time and snapshot info added later to decide whether to run compaction or not for the users. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722719791


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStats(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null
+          || ((long) toRecord.get(Column.LAST_UPDATED_AT.ordinal())
+              < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+        toRecord.set(
+            Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+        toRecord.set(
+            Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+            fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+      }
+    }
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION.name()).type().asStructType();
+    Record partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(Column.PARTITION.ordinal(), partitionData);
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), 0L);
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 0);
+        record.set(Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.ordinal(), 0L);
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // Note: Not computing the `TOTAL_RECORD_COUNT` for now as it needs scanning the data.

Review Comment:
   when the position or equality deletes are not present, we can just use `DATA_RECORD_COUNT` field as it will be same as expected `TOTAL_RECORD_COUNT` for the given partition value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722735848


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests
+  private static final int PARTITION_COUNT = 10000;
+
+  // one data file manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 1;

Review Comment:
   It takes lot of time to generate manifests (for 10k manifests it took around 20 minutes). 
   I will optimize it for a balanced number and user can increase the counters if they want to see benchmarks against more manifests with less data files or less manifests with more data file per manifests. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1723533548


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsWriterUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // create 3 manifests with same partition values
+  private static final int ITERATION_COUNTER = 3;
+
+  // create 3 * 10000 manifests

Review Comment:
   Iteration probably shouldn't be used as a name here since it conflates with the benchmark itself. I would look for another  variable name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2301167087

   Thanks @RussellSpitzer for the review. I have updated the comment. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697840726


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+  private String branch;
+
+  public GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  public GeneratePartitionStats(Table table, String branch) {
+    this.table = table;
+    this.branch = branch;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    Snapshot currentSnapshot = SnapshotUtil.latestSnapshot(table, branch);
+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;
+    }
+
+    Types.StructType partitionType = Partitioning.partitionType(table);
+    // Map of partitionData, partition-stats-entry per partitionData.
+    // Sorting the records based on partition as per spec.
+    Map<Record, Record> partitionEntryMap =
+        new ConcurrentSkipListMap<>(Comparators.forType(partitionType));

Review Comment:
   I also tried ConcurrentSkipListMap, but it seems its `compute` and `merge` methods are not guaranteed to be atomic, which can lead to consistency issues. So I guess we need to sort the records after we populate the map.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698068822


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag

Review Comment:
   True. Maybe we didn't think well about this field during spec design. We can drop a todo and make this field deprecated if required later on. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698116352


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");

Review Comment:
   This is unified partitionType. If the table is currently unpartitioned but it was partitioned before, we still allow computing stats. 
   
   Yes. If the table is never partitioned, we are not allowing computing stats because it is of no use (or it is not a partition stats anymore). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698404347


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {

Review Comment:
   Since I started as a spark action, used verb. Will change it to noun. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571025684


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))
+            .build();
+    DataWriter<Record> writer =
+        factory.newDataWriter(
+            EncryptedFiles.encryptedOutput(outputFile, EncryptionKeyMetadata.EMPTY),
+            PartitionSpec.unpartitioned(),
+            null);
+    try (Closeable toClose = writer) {
+      records.forEachRemaining(writer::write);
+    } catch (IOException e) {
+      throw new UncheckedIOException(e);
+    }
+  }
+
+  public static CloseableIterable<Record> readPartitionStatsFile(
+      Schema schema, InputFile inputFile) {
+    // TODO: support other formats or introduce GenericFileReader

Review Comment:
   we don't have `GenericFileReader`, all reading is based on tableScan currently. For this we may need `GenericFileReader` similar to the writer we used



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1692581100


##########
data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneratePartitionStats {
+  private static final Logger LOG = LoggerFactory.getLogger(GeneratePartitionStats.class);
+
+  private final Table table;
+
+  GeneratePartitionStats(Table table) {
+    this.table = table;
+  }
+
+  /**
+   * Computes the partition stats for the current snapshot and writes it into the metadata folder.
+   *
+   * @return {@link PartitionStatisticsFile} for the latest snapshot id or null if table doesn't
+   *     have any snapshot.
+   */
+  public PartitionStatisticsFile generate() {
+    // TODO: Support an option to accept branches and tag info and lookup corresponding snapshot.
+    Snapshot currentSnapshot = table.currentSnapshot();
+    if (currentSnapshot == null) {
+      return null;
+    }
+
+    // Map of partitionData, partition-stats-entry per partitionData
+    Map<Record, Record> partitionEntryMap = Maps.newConcurrentMap();
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    List<ManifestFile> manifestFiles = currentSnapshot.allManifests(table.io());
+    Tasks.foreach(manifestFiles)
+        .suppressFailureWhenFinished()
+        .executeWith(ThreadPools.getWorkerPool())
+        .onFailure(
+            (file, thrown) ->
+                LOG.warn(

Review Comment:
   stopping on failure and throwing exception now. Still keeping warning to print the file path (like other places of code)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2245198784

   @RussellSpitzer: Thanks for the review. 
   I have addressed all the comments and added the mentioned tests. I have plan to add some more tests tomorrow (related to delete file stats)
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1724415060


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ImmutableGenericPartitionStatisticsFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.PartitionStatisticsFile;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PartitionStatsGenerator {

Review Comment:
   Updated the javadocs. 
   
   `PartitionStatsGenerator` is to generate the stats file using util classes. Util class is a helper for that.  
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1728009901


##########
data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.apache.iceberg.data.PartitionStatsGenerator;
+import org.apache.iceberg.data.PartitionStatsGeneratorUtil;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@Fork(1)
+@State(Scope.Benchmark)
+@Warmup(iterations = 2)
+@Measurement(iterations = 5)
+@Timeout(time = 1000, timeUnit = TimeUnit.HOURS)
+@BenchmarkMode(Mode.SingleShotTime)
+public class PartitionStatsGeneratorBenchmark {
+
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "c1", Types.IntegerType.get()),
+          optional(2, "c2", Types.StringType.get()),
+          optional(3, "c3", Types.StringType.get()));
+
+  protected static final PartitionSpec SPEC =
+      PartitionSpec.builderFor(SCHEMA).identity("c1").build();
+
+  private String baseDir;
+
+  // Create 10k manifests
+  private static final int MANIFEST_COUNTER = 10000;
+
+  // each manifest with 100 partition values
+  private static final int PARTITION_PER_MANIFEST = 100;
+
+  // 20 data files per partition, which results in 2k data files per manifest
+  private static final int DATA_FILES_PER_PARTITION_COUNT = 20;
+
+  private Table table;
+
+  @Setup
+  public void setupBenchmark() {
+    baseDir =
+        Paths.get(new File(System.getProperty("java.io.tmpdir")).getAbsolutePath()).toString();
+    table = TestTables.create(new File(baseDir), "foo", SCHEMA, SPEC, SortOrder.unsorted(), 2);
+
+    IntStream.range(0, MANIFEST_COUNTER)

Review Comment:
   It would be nice to parallelize the preparation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2285486800

   ping @aokolnychyi, @RussellSpitzer  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714692537


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(

Review Comment:
   Good suggestion. Optimized it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714682727


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.IdentityPartitionConverters;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a schema that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    Preconditions.checkState(
+        !partitionType.fields().isEmpty(), "getting schema for an unpartitioned table");
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),

Review Comment:
   The intention is to indicate the latest spec id that has modified this partition value. 
   
   After the partition evolution, even though we use unified tuple, if the partition is not modified by the latest spec id, It will still be remained as old spec id value. 
   
   This can help in knowing how many partitions have the new data inserted as per new spec and how many partition might need to call rewrite data files (to convert them to new spec)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721885650


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a schema as per partition statistics spec based on the given partition type.

Review Comment:
   Nit here and other places in the doc, the doc string is probably clearer as 
   "Generates the Partition Stats Files Schema based on a given a partition type" 
   
   Mostly saying "generates a schema" or "returns a schema" don't add much info. The important part is that it provides the schema for a Partition Stats File
   
   Should we also note this should be called with the unified spec? Or is it ok to use with a non-unified spec?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721975924


##########
core/src/main/java/org/apache/iceberg/data/IdentityPartitionConverters.java:
##########
@@ -48,6 +50,13 @@ public static Object convertConstant(Type type, Object value) {
       case FIXED:
         if (value instanceof GenericData.Fixed) {
           return ((GenericData.Fixed) value).bytes();
+        } else if (value instanceof ByteBuffer) {
+          return ((ByteBuffer) value).array();

Review Comment:
   Use  
   ```java
   ByteBuffers.toByteArray
   ```
   https://github.com/apache/iceberg/blob/c07f2aabc0a1d02f068ecf1514d2479c0fbdd3b0/api/src/main/java/org/apache/iceberg/util/ByteBuffers.java#L27
   
   A safer option (same suggestion for UUID)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2307890941

   I took a detailed look at the implementation/spec today. Here are my high-level points:
   
   - The local algorithm was the right choice. I am on board with using `ConcurrentHashMap` and coerced partitions as keys.
   - The logic is split between 3 utility classes that are hard to navigate.
   - In my view, `Record` is not the best abstraction to expose when computing stats in `core`.
       - Projecting and getting values from `Record` is very fragile.
       - It is necessary to have `Record` to use generic writers but `PartitionStatsUtil` is in `core` and is supposed to be helpful for arbitrary implementations, not only those that rely on generic writers.
       - The `fromManifest` method in `PartitionStatsUtil` does not return a valid `Record` as it doesn’t transform values (it shouldn’t, the writer has to do that). We need a better abstraction in `core`.
   
   I'd consider restructuring the code as below (names are suggestions):
   
   1. Add a wrapper class to represent partition stats in `core`.
   
   ```
   public class PartitionStats {
   
     private final int specId;
     private final StructLike partition;
     ...
     private Long lastUpdatedSnapshotId = null;
   
     public void liveEntry(ContentFile<?> file, Snapshot snapshot) {
       // move the existing logic for updating stats from live entires here
     }
   
     public void deletedEntry(Snapshot snapshot) {
       if (snapshot != null) {
         // update last modified info to have accurate stats
       }
     }
   }
   ```
   
   2. Keep `PartitionStatsUtil` in `core` but change its purpose. It does everything except writing.
   
   ```
   public class PartitionStatsUtil {
   
     public static final int PARTITION_FIELD_ID = 1;
     public static final String PARTITION_FIELD_NAME = "partition";
     public static final NestedField SPEC_ID = required(2, "spec_id", IntegerType.get());
     ...
    
     public static Schema schema(StructType partitionType) {
       // return the schema of records in partition stats files
     }
   
     public static Iterable<PartitionStats> computeStats(Table table, Snapshot snapshot) {
       // compute stats like in this PR using ConcurrentHashMap
       // don't convert ManifestEntry to PartitionStats, invoke liveEntry/deletedEntry to update stats as needed
     }
   
     public static List<PartitionStats> sortStats(Iterable<PartitionStats> stats, StructType partitionType) {
       // sort stats per spec if they have to be written (having this separately is optional)
     }
   }
   ```
   
   3. Add `PartitionStatsWriter` that would wrap `PartitionStats` as `Record` internally and write using generic writers (like in this PR). The writer will be responsible for converting values using `IdentityPartitionConverters`.
   
   What do you think, @ajantha-bhat?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679683800


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null

Review Comment:
   Same question as below, shouldn't we set toUpdate to the value of fromRecord if toUpdate is null but fromRecord is not?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679828029


##########
data/src/test/java/org/apache/iceberg/data/TestPartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.PartitionStatsUtil.Column;
+
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Random;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TestTables;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.assertj.core.api.Assertions;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestPartitionStatsWriterUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(TestPartitionStatsWriterUtil.class);
+
+  private static final Schema SCHEMA =
+      new Schema(
+          Types.NestedField.required(1, "id", Types.LongType.get()),
+          Types.NestedField.optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional(3, "binary", Types.BinaryType.get()));
+
+  @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testPartitionStats() throws Exception {
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("id").build();
+    Table testTable =
+        TestTables.create(
+            temp.newFolder("test_partition_stats"),
+            "test_partition_stats",
+            SCHEMA,
+            spec,
+            SortOrder.unsorted(),
+            2);
+
+    Types.StructType partitionSchema = Partitioning.partitionType(testTable);
+    Schema dataSchema = PartitionStatsUtil.schema(partitionSchema);
+
+    ImmutableList.Builder<Record> partitionListBuilder = ImmutableList.builder();
+
+    long seed = System.currentTimeMillis();
+    LOG.info("Seed used for random generator is {}", seed);

Review Comment:
   not sure this will be useful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679727820


##########
data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionStatsUtil;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.encryption.EncryptedFiles;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.types.Types;
+
+public final class PartitionStatsWriterUtil {
+
+  private PartitionStatsWriterUtil() {}
+
+  public static OutputFile newPartitionStatsFile(Table table, long snapshotId) {
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)
+                .operations()
+                .metadataFileLocation(
+                    fileFormat.addExtension(String.format("partition-stats-%d", snapshotId))));
+  }
+
+  public static void writePartitionStatsFile(
+      Table table, Iterator<Record> records, OutputFile outputFile) {
+    Schema dataSchema = PartitionStatsUtil.schema(Partitioning.partitionType(table));
+    FileFormat fileFormat =
+        FileFormat.fromString(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
+    FileWriterFactory<Record> factory =
+        GenericFileWriterFactory.builderFor(table)
+            .dataSchema(dataSchema)
+            .dataFileFormat(fileFormat)
+            .dataSortOrder(sortOrder(dataSchema))

Review Comment:
   Yeah I'm not sure this actually has any use, not sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1679653579


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries =
+        CloseableIterable.transform(
+            ManifestFiles.open(manifest, table.io(), table.specs())
+                .select(scanColumns(manifest.content())) // don't select stats columns
+                .liveEntries(),
+            t ->
+                (ManifestEntry<? extends ContentFile<?>>)
+                    // defensive copy of manifest entry without stats columns
+                    t.copyWithoutStats());
+
+    return CloseableIterable.transform(
+        entries, entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  public static void updateRecord(Record toUpdate, Record fromRecord) {
+    toUpdate.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toUpdate.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    incrementLong(toUpdate, fromRecord, Column.DATA_RECORD_COUNT);
+    incrementInt(toUpdate, fromRecord, Column.DATA_FILE_COUNT);
+    incrementLong(toUpdate, fromRecord, Column.DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toUpdate, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toUpdate, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (toUpdate.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null
+        && ((long) toUpdate.get(Column.LAST_UPDATED_AT.ordinal())
+            < (long) fromRecord.get(Column.LAST_UPDATED_AT.ordinal()))) {
+      toUpdate.set(
+          Column.LAST_UPDATED_AT.ordinal(), fromRecord.get(Column.LAST_UPDATED_AT.ordinal()));
+      toUpdate.set(
+          Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(),
+          fromRecord.get(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal()));
+    }
+  }
+
+  public static Record partitionDataToRecord(
+      Types.StructType partitionSchema, PartitionData partitionData) {
+    GenericRecord genericRecord = GenericRecord.create(partitionSchema);
+    for (int index = 0; index < partitionData.size(); index++) {
+      genericRecord.set(index, partitionData.get(index));
+    }
+
+    return genericRecord;
+  }
+
+  private static Record fromManifestEntry(
+      ManifestEntry<?> entry, Table table, Schema recordSchema) {
+    GenericRecord record = GenericRecord.create(recordSchema);
+    Types.StructType partitionType =
+        recordSchema.findField(Column.PARTITION_DATA.name()).type().asStructType();
+    PartitionData partitionData = coercedPartitionData(entry.file(), table.specs(), partitionType);
+    record.set(
+        Column.PARTITION_DATA.ordinal(), partitionDataToRecord(partitionType, partitionData));
+    record.set(Column.SPEC_ID.ordinal(), entry.file().specId());
+
+    Snapshot snapshot = table.snapshot(entry.snapshotId());
+    if (snapshot != null) {
+      record.set(Column.LAST_UPDATED_SNAPSHOT_ID.ordinal(), snapshot.snapshotId());
+      record.set(Column.LAST_UPDATED_AT.ordinal(), snapshot.timestampMillis());
+    }
+
+    switch (entry.file().content()) {
+      case DATA:
+        record.set(Column.DATA_FILE_COUNT.ordinal(), 1);
+        record.set(Column.DATA_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        record.set(Column.DATA_FILE_SIZE_IN_BYTES.ordinal(), entry.file().fileSizeInBytes());
+        break;
+      case POSITION_DELETES:
+        record.set(Column.POSITION_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.POSITION_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      case EQUALITY_DELETES:
+        record.set(Column.EQUALITY_DELETE_FILE_COUNT.ordinal(), 1);
+        record.set(Column.EQUALITY_DELETE_RECORD_COUNT.ordinal(), entry.file().recordCount());
+        break;
+      default:
+        throw new UnsupportedOperationException(
+            "Unsupported file content type: " + entry.file().content());
+    }
+
+    // TODO: optionally compute TOTAL_RECORD_COUNT based on the flag
+    return record;
+  }
+
+  private static PartitionData coercedPartitionData(
+      ContentFile<?> file, Map<Integer, PartitionSpec> specs, Types.StructType partitionType) {
+    // keep the partition data as per the unified spec by coercing
+    StructLike partition =
+        PartitionUtil.coercePartition(partitionType, specs.get(file.specId()), file.partition());
+    PartitionData data = new PartitionData(partitionType);
+    for (int i = 0; i < partitionType.fields().size(); i++) {
+      Object val = partition.get(i, partitionType.fields().get(i).type().typeId().javaClass());
+      if (val != null) {
+        data.set(i, val);
+      }
+    }
+    return data;
+  }
+
+  private static List<String> scanColumns(ManifestContent content) {
+    switch (content) {
+      case DATA:
+        return BaseScan.SCAN_COLUMNS;
+      case DELETES:
+        return BaseScan.DELETE_SCAN_COLUMNS;
+      default:
+        throw new UnsupportedOperationException("Cannot read unknown manifest type: " + content);
+    }
+  }
+
+  private static void incrementLong(Record toUpdate, Record fromRecord, Column column) {
+    toUpdate.set(
+        column.ordinal(),
+        (long) toUpdate.get(column.ordinal()) + (long) fromRecord.get(column.ordinal()));

Review Comment:
   Since record is structlike we can do get(column.ordinal(), Long.class)
   might look a little cleaner



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2271191550

   @aokolnychyi , @RussellSpitzer, @lirui-apache: Please take another look at the PR. I have addressed all the comments.   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698436823


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    TOTAL_DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  /**
+   * Generates a Schema object as per partition statistics spec based on the given partition type.
+   *
+   * @param partitionType the struct type that defines the structure of the partition.
+   * @return a Schema object that corresponds to the provided partition type.
+   */
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(
+            5, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            6, Column.POSITION_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            7, Column.POSITION_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(
+            8, Column.EQUALITY_DELETE_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            9, Column.EQUALITY_DELETE_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.optional(10, Column.TOTAL_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.optional(11, Column.LAST_UPDATED_AT.name(), Types.LongType.get()),
+        Types.NestedField.optional(
+            12, Column.LAST_UPDATED_SNAPSHOT_ID.name(), Types.LongType.get()));
+  }
+
+  /**
+   * Creates an iterable of partition stats records from a given manifest file, using the specified
+   * table and record schema.
+   *
+   * @param table the table from which the manifest file is derived.
+   * @param manifest the manifest file containing metadata about the records.
+   * @param recordSchema the schema defining the structure of the records.
+   * @return a CloseableIterable of partition stats records as defined by the manifest file and
+   *     record schema.
+   */
+  public static CloseableIterable<Record> fromManifest(
+      Table table, ManifestFile manifest, Schema recordSchema) {
+    return CloseableIterable.transform(
+        ManifestFiles.open(manifest, table.io(), table.specs())
+            .select(BaseScan.scanColumns(manifest.content()))
+            .liveEntries(),
+        entry -> fromManifestEntry(entry, table, recordSchema));
+  }
+
+  /**
+   * Appends statistics from one Record to another.
+   *
+   * @param toRecord the Record to which statistics will be appended.
+   * @param fromRecord the Record from which statistics will be sourced.
+   */
+  public static void appendStatsFromRecord(Record toRecord, Record fromRecord) {
+    Preconditions.checkState(toRecord != null, "Record to update cannot be null");
+    Preconditions.checkState(fromRecord != null, "Record to update from cannot be null");
+
+    toRecord.set(
+        Column.SPEC_ID.ordinal(),
+        Math.max(
+            (int) toRecord.get(Column.SPEC_ID.ordinal()),
+            (int) fromRecord.get(Column.SPEC_ID.ordinal())));
+    checkAndIncrementLong(toRecord, fromRecord, Column.DATA_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.DATA_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_DATA_FILE_SIZE_IN_BYTES);
+    checkAndIncrementLong(toRecord, fromRecord, Column.POSITION_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.POSITION_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.EQUALITY_DELETE_RECORD_COUNT);
+    checkAndIncrementInt(toRecord, fromRecord, Column.EQUALITY_DELETE_FILE_COUNT);
+    checkAndIncrementLong(toRecord, fromRecord, Column.TOTAL_RECORD_COUNT);
+    if (fromRecord.get(Column.LAST_UPDATED_AT.ordinal()) != null) {
+      if (toRecord.get(Column.LAST_UPDATED_AT.ordinal()) == null

Review Comment:
   > Hm, can we actually compute last_updated_at if at least one side has null? 
   
   `fromRecord` side if it is null, then that snapshot might have expired and older than `toRecord` side. So, it won't be latest. Hence, no need to update. 
   
   If `toRecord` side is null then `fromRecord` might be latest hence updating. 
   
   So, when we go through all records for that partition, it will have the latest snapshot info. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1698066417


##########
core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+
+public class PartitionStatsUtil {
+
+  private PartitionStatsUtil() {}
+
+  public enum Column {
+    PARTITION_DATA,
+    SPEC_ID,
+    DATA_RECORD_COUNT,
+    DATA_FILE_COUNT,
+    DATA_FILE_SIZE_IN_BYTES,
+    POSITION_DELETE_RECORD_COUNT,
+    POSITION_DELETE_FILE_COUNT,
+    EQUALITY_DELETE_RECORD_COUNT,
+    EQUALITY_DELETE_FILE_COUNT,
+    TOTAL_RECORD_COUNT,
+    LAST_UPDATED_AT,
+    LAST_UPDATED_SNAPSHOT_ID
+  }
+
+  public static Schema schema(Types.StructType partitionType) {
+    if (partitionType.fields().isEmpty()) {
+      throw new IllegalArgumentException("getting schema for an unpartitioned table");
+    }
+
+    return new Schema(
+        Types.NestedField.required(1, Column.PARTITION_DATA.name(), partitionType),
+        Types.NestedField.required(2, Column.SPEC_ID.name(), Types.IntegerType.get()),
+        Types.NestedField.required(3, Column.DATA_RECORD_COUNT.name(), Types.LongType.get()),
+        Types.NestedField.required(4, Column.DATA_FILE_COUNT.name(), Types.IntegerType.get()),
+        Types.NestedField.required(5, Column.DATA_FILE_SIZE_IN_BYTES.name(), Types.LongType.get()),
+        Types.NestedField.optional(

Review Comment:
   last computed snapshot may be unavailable at the time of computing the stats (due to expire snapshots). So, `LAST_UPDATED_SNAPSHOT_ID` and `LAST_UPDATED_AT` can be null. Hence, it is optional. Also, it doesn't make sense to fill it with 0 as snapshot id 0 is not meaningful in iceberg.
   
   To maintain uniform behaviour of all optional fields, the stats like `TOTAL_RECORD_COUNT, EQUALITY_DELETE_FILE_COUNT, EQUALITY_DELETE_RECORD_COUNT, POSITION_DELETE_FILE_COUNT, POSITION_DELETE_RECORD_COUNT` will be made null when absent instead of 0.
   
   I have added a nice testcase which validates all these now. 
   
   Also, schema(optional & required) is as spec: https://iceberg.apache.org/spec/#partition-statistics-file
   If we decide to change optional to required now, it has to go through vote :(



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Add a util to read write partition stats [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10176:
URL: https://github.com/apache/iceberg/pull/10176#discussion_r1741651006


##########
core/src/main/java/org/apache/iceberg/data/PartitionStatsRecord.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.PartitionStats;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.StructType;
+
+public class PartitionStatsRecord implements Record, StructLike {
+  private static final LoadingCache<StructType, Map<String, Integer>> NAME_MAP_CACHE =

Review Comment:
   Similar to existing `GenericRecord` class, but it wraps the existing `PartitionStats` and used only for partition stats



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org