You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2025/11/03 10:33:38 UTC

[PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

jbonofre opened a new pull request, #12897:
URL: https://github.com/apache/iceberg/pull/12897

   Following the update on the spec regarding `source-id` and `source-ids` (thanks again @Fokko πŸ˜„ ), here's the PR to introduce `source-ids` field in partition spec.
   
   A few notes:
   1. Internal representation is now based on `source-ids`
   2. Serialization/deserialization supports `source-id` and `source-ids` elements in the json (exclusive), both populating `source-ids` internal representation (as `List<Integer>`)
   3. The `TestPartitionSpecParser` is still testing `source-id`, but also `source-ids` parsing, and neither `source-id` and `source-ids` presence (throwing `IllegalArgumentException` in that case)
   4. For backward compatibility (especially for the transforms), source id is still supported (using the first element in the internal representation) and some methods have been flagged as deprecated to encourage use of source ids.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060368165


##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   Yes, that's an interesting point, thanks for bringing this πŸ˜„. Let's see what the others are thinking here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2065540144


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -625,7 +634,7 @@ PartitionSpec buildUnchecked() {
 
   static void checkCompatibility(PartitionSpec spec, Schema schema) {
     for (PartitionField field : spec.fields) {
-      Type sourceType = schema.findType(field.sourceId());
+      Type sourceType = schema.findType(field.sourceIds().get(0));

Review Comment:
   Yup. I'm doing a new update. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2093531700


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -625,7 +634,7 @@ PartitionSpec buildUnchecked() {
 
   static void checkCompatibility(PartitionSpec spec, Schema schema) {
     for (PartitionField field : spec.fields) {
-      Type sourceType = schema.findType(field.sourceId());
+      Type sourceType = schema.findType(field.sourceIds().get(0));

Review Comment:
   I introduced new `canTransform(List)` and `getResultType(List)` to support multi-args in transforms.
   I didn't remove the "old" methods, and I propose to deprecate it.
   Thoughts ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12897: Core: Implement source-ids to deal with multi arguments transforms
URL: https://github.com/apache/iceberg/pull/12897


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060142013


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -70,14 +100,14 @@ public boolean equals(Object other) {
     }
 
     PartitionField that = (PartitionField) other;
-    return sourceId == that.sourceId
+    return sourceIds.toString().equals(that.sourceIds.toString())

Review Comment:
   Just wondering why not `Objects.equals(sourceIds, that.sourceIds);` do we need the extra string conversion? 



##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   Doesn't this breaks backward compatibility? and this is not allowed for minor versions.
   
   We need to implement custom `writeObject` and `readObject`  to ensure we can serialize the field if it has only `sourceId`?
   
   ```
   private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException {
       in.defaultReadObject(); // read all default fields
   
       // Ensure backward compatibility
       if (sourceIds == null && sourceId != null) {
           sourceIds = Collections.singletonList(sourceId);
       }
   }
   
   private void writeObject(ObjectOutputStream out) throws IOException {
       out.defaultWriteObject(); // only write sourceIds
   }
   
   ```
   
   Note: I know we have handled the JSON serialization properly in `PartitionSpecParser`. Just wondering about default java serializer here.
   



##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -98,7 +101,16 @@ static void toJsonFields(UnboundPartitionSpec spec, JsonGenerator generator) thr
       generator.writeStartObject();
       generator.writeStringField(NAME, field.name());
       generator.writeStringField(TRANSFORM, field.transformAsString());
-      generator.writeNumberField(SOURCE_ID, field.sourceId());
+      if (field.sourceIds().size() == 1) {
+        generator.writeNumberField(SOURCE_ID, field.sourceId());
+      } else {
+        generator.writeFieldName(SOURCE_IDS);
+        generator.writeStartArray();
+        for (Integer value : field.sourceIds()) {
+          generator.writeNumber(value);
+        }
+        generator.writeEndArray();
+      }

Review Comment:
   nit: Iceberg follows the new line spacing for code blocks 
   https://iceberg.apache.org/contribute/?h=contribut#block-spacing



##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   How does this work for multi arg partition field? 
   Do we need a logic that accepts the `sourceIds` and resolve the name of multi arg transform? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2908573611


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########


Review Comment:
   This is broken now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2931241498


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -58,7 +74,10 @@ public String name() {
 
   @Override
   public String toString() {
-    return fieldId + ": " + name + ": " + transform + "(" + sourceId + ")";
+    if (sourceIds.size() == 1) {
+      return fieldId + ": " + name + ": " + transform + "(" + sourceIds.get(0) + ")";
+    }
+    return fieldId + ": " + name + ": " + transform + "(" + sourceIds + ")";

Review Comment:
   Done, good catch ! Thanks !



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061141617


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   OK, let me refactore this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060237024


##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   I think we only use JSON serialization. I don't think we are using Java serialization. Do we ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060447565


##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   I think we are fine here, we don't expect a client/server using this code to be on different library versions.  I believe this currently is hit when using Spark because we are going to serialize the table metadata around. For usages like this I don't want to have to support multiple versions on driver / executor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2887451639

   I did a first update to introduce multi-args in `Transform`. I will check/update the tests too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2931245431

   I fixed the tests. I'm addressing the pending comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-4920230153

   Hi @jbonofre, are you still planning to pursue this PR? I'm asking because I recently implemented the v3 read-tolerance side of this in PyIceberg (apache/iceberg-python#3630: fields with multiple `source-ids` parse and are treated as unknown transforms, with spec-compliant `source-id`/`source-ids` round-trip), and I'd like to help bring the Java implementation to that baseline β€” today, metadata containing a `source-ids` field cannot be read at all. Happy to help here if that's useful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060211621


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -70,14 +100,14 @@ public boolean equals(Object other) {
     }
 
     PartitionField that = (PartitionField) other;
-    return sourceId == that.sourceId
+    return sourceIds.toString().equals(that.sourceIds.toString())

Review Comment:
   Yeah, either way is fine. Let me do the change if you prefer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060507585


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated
   PartitionField(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-    this.sourceId = sourceId;
+    this(List.of(sourceId), fieldId, name, transform);
+  }
+
+  PartitionField(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+    this.sourceIds = sourceIds;
     this.fieldId = fieldId;
     this.name = name;
     this.transform = transform;
   }
 
-  /** Returns the field id of the source field in the {@link PartitionSpec spec's} table schema. */
+  /**
+   * Returns the source field in the {@link PartitionSpec spec's} table schema.
+   *
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use sourceIds() instead.
+   */
+  @Deprecated
   public int sourceId() {
-    return sourceId;
+    Preconditions.checkArgument(sourceIds.size() == 1, "Only one source is supported");

Review Comment:
   Better yeah. Let me update. Thanks for the proposal !



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061143410


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +147,29 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
+      Preconditions.checkArgument(
+          element.has(SOURCE_ID) || element.has(SOURCE_IDS),
+          "Cannot parse partition field, either "
+              + SOURCE_ID
+              + " or "
+              + SOURCE_IDS
+              + " has to be present");

Review Comment:
   `source-id` and `source-ids` are exclusive, meaning that if `source-id` is present, `source-ids` should not be there, and if `source-ids` is present `source-id` should not be there.
   I'm adding a check to raise to check if not both are provided.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060493653


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   The current logic looks like
   ```java
   if (check conflicts). {
     if identity {
       make sure we aren't using a different field name for this identity transform
     } else {
      make sure we aren't matching any other column name
     }
   }
   Make sure it's not empty
   Make sure we haven't already used this name for another partition
   Add partition
   ```
   
   I have no idea why those last 2 checks aren't part of the "if check conflicts" branch
   
   Anyyyyyyway. I think this whole validation probably should be rewritten. The first branch we are checking basically based a lot of implicit assumptions when we should just be passing in the transform. But we don't have to do any of that now. 
   
   For now I think we should pass in sourceIds and just have the first branch include a "if sourceIds.length == 1"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060504659


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -54,17 +55,17 @@ private PartitionSpec.Builder copyToBuilder(Schema schema) {
     PartitionSpec.Builder builder = PartitionSpec.builderFor(schema).withSpecId(specId);
 
     for (UnboundPartitionField field : fields) {
-      Type fieldType = schema.findType(field.sourceId);
+      Type fieldType = schema.findType(field.sourceIds.get(0));

Review Comment:
   I know we don't have a multi-arg transform to resolve here yet but this doesn't seem like the right thing to do. I think Transforms.fromString needs to be modified to accept fromString(list[types], transformName)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060214278


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   The sourceId here is to resolve conflict (no impact on name):
   
   ```
           if (sourceColumnId != null) {
             // for identity transform case we allow conflicts between partition and schema field name
             // as
             //   long as they are sourced from the same schema field
             Preconditions.checkArgument(
                 schemaField == null || schemaField.fieldId() == sourceColumnId,
                 "Cannot create identity partition sourced from different field in schema: %s",
                 name);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2161244726


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -636,15 +648,20 @@ static void checkCompatibility(PartitionSpec spec, Schema schema) {
       // checks
       if (!transform.equals(Transforms.alwaysNull())) {
         ValidationException.check(
-            sourceType != null, "Cannot find source column for partition field: %s", field);
-        ValidationException.check(
-            sourceType.isPrimitiveType(),
-            "Cannot partition by non-primitive source field: %s",
-            sourceType);
+            !sourceTypes.isEmpty(), "Cannot find source column for partition field: %s", field);
+        for (Type sourceType : sourceTypes) {
+          ValidationException.check(
+              sourceType != null, "Cannot find source column for partition field: %s", field);
+          ValidationException.check(
+              sourceType.isPrimitiveType(),
+              "Cannot partition by non-primitive source field: %s (%s)",
+              field,
+              sourceType);
+        }

Review Comment:
   nit: newline



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "sfc-gh-rspitzer (via GitHub)" <gi...@apache.org>.
sfc-gh-rspitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2908538079


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,42 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
   PartitionField(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-    this.sourceId = sourceId;
+    this(ImmutableList.of(sourceId), fieldId, name, transform);
+  }
+
+  PartitionField(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+    this.sourceIds = ImmutableList.copyOf(sourceIds);
     this.fieldId = fieldId;
     this.name = name;
     this.transform = transform;
   }
 
   /** Returns the field id of the source field in the {@link PartitionSpec spec's} table schema. */
   public int sourceId() {
-    return sourceId;
+    return sourceIds.get(0);

Review Comment:
   I still think there should be a precondition that sourceIds length is = 1,  just to make sure folk are using this correctly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2908553196


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -58,7 +74,10 @@ public String name() {
 
   @Override
   public String toString() {
-    return fieldId + ": " + name + ": " + transform + "(" + sourceId + ")";
+    if (sourceIds.size() == 1) {
+      return fieldId + ": " + name + ": " + transform + "(" + sourceIds.get(0) + ")";
+    }
+    return fieldId + ": " + name + ": " + transform + "(" + sourceIds + ")";

Review Comment:
   The multi-source-id branch uses `List.toString()` directly, which produces square brackets in the output: `transform([1, 2, 3])` instead of `transform(1, 2, 3)`.
   
   Using a `Joiner` (as done elsewhere in the api module, e.g. `UnboundPredicate`, `BoundSetPredicate`) would fix this and also eliminate the need for the `size() == 1` special case:
   
   ```java
   private static final Joiner COMMA = Joiner.on(", ");
   
   @Override
   public String toString() {
     return fieldId + ": " + name + ": " + transform + "(" + COMMA.join(sourceIds) + ")";
   }
   ```
   
   Same applies to `SortField.toString()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2931276818


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +144,37 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
 
-      // partition field ids are missing in old PartitionSpec, they always auto-increment from
-      // PARTITION_DATA_ID_START
-      if (element.has(FIELD_ID)) {
-        builder.addField(transform, sourceId, JsonUtil.getInt(FIELD_ID, element), name);
-        fieldIdCount++;
+      if (element.has(SOURCE_IDS)) {

Review Comment:
   I'm hesitating here. To limit the impact, finally, I preferred to log (warn) when both source-ids and source-id are present (instead of failing).
   
   Thoughts ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3465461015

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2095566084


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -602,17 +602,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {

Review Comment:
   Do we need a preconditions against null sourceIds in the list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060368165


##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   Yes, that's an interesting point, thanks for bringing this πŸ˜„ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060514306


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +147,29 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
+      Preconditions.checkArgument(
+          element.has(SOURCE_ID) || element.has(SOURCE_IDS),
+          "Cannot parse partition field, either "
+              + SOURCE_ID
+              + " or "
+              + SOURCE_IDS
+              + " has to be present");

Review Comment:
   One more check I think we need here, if both are present they must match, currently the code just always uses sourceId if it's present even if sourceIds is also present and different.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060514306


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +147,29 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
+      Preconditions.checkArgument(
+          element.has(SOURCE_ID) || element.has(SOURCE_IDS),
+          "Cannot parse partition field, either "
+              + SOURCE_ID
+              + " or "
+              + SOURCE_IDS
+              + " has to be present");

Review Comment:
   One more check I think we need here, if both are present they must match



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2830855584

   Thanks @RussellSpitzer ! Let me fix the util methods and add tests. Thanks again !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061143136


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecParser.java:
##########
@@ -95,6 +96,56 @@ public void testFromJsonWithFieldId() {
     assertThat(spec.fields().get(1).fieldId()).isEqualTo(1000);
   }
 
+  @TestTemplate
+  public void testFromJsonWithSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"source-ids\" : [ 1, 2 ],\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"source-ids\" : [ 2, 3 ],\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    PartitionSpec spec = PartitionSpecParser.fromJson(table.schema(), specString);
+
+    assertThat(spec.fields()).hasSize(2);
+    assertThat(spec.fields().get(0).sourceIds()).hasSize(2);
+    assertThat(spec.fields().get(1).sourceIds()).hasSize(2);
+  }
+
+  @TestTemplate
+  public void testFromJsonWithoutSourceIdAndSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    assertThatThrownBy(
+            () -> {
+              PartitionSpecParser.fromJson(table.schema(), specString);
+            })
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage(
+            "Cannot parse partition field, either source-id or source-ids has to be present");
+  }

Review Comment:
   I added a test in `TestPartitionSpecParser` testing when neither `source-id` and `source-ids` are provided: `testFromJsonWithoutSourceIdAndSourceIds()`.
   I will add additional tests regarding your previous mention.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12897: Core: Implement source-ids to deal with multi arguments transforms
URL: https://github.com/apache/iceberg/pull/12897


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2931262948


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########


Review Comment:
   I used a simple approach now, iterating on all source ids:
   
   ```
     for (PartitionField field : fields) {
         for (int sourceId : field.sourceIds()) {
             multiMap.put(sourceId, field);
         }
     }
   ```
   
   I think it makes sense in this context.
   
   Thoughts ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060511995


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -118,7 +128,12 @@ public String transformAsString() {
     }
 
     public int sourceId() {
-      return sourceId;
+      Preconditions.checkArgument(sourceIds.size() == 1, "Only one source is expected");

Review Comment:
   cannot call sourceId() on a transform with multiple arguments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060508764


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -54,17 +55,17 @@ private PartitionSpec.Builder copyToBuilder(Schema schema) {
     PartitionSpec.Builder builder = PartitionSpec.builderFor(schema).withSpecId(specId);
 
     for (UnboundPartitionField field : fields) {
-      Type fieldType = schema.findType(field.sourceId);
+      Type fieldType = schema.findType(field.sourceIds.get(0));

Review Comment:
   Yes, agree. Let me update that. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2829875290

   @rdblue @Fokko @RussellSpitzer ^^ Thanks !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060214278


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   The sourceId here is to resolve conflict (no impact on name):
   
   ```
           if (sourceColumnId != null) {
             // for identity transform case we allow conflicts between partition and schema field name
             // as
             //   long as they are sourced from the same schema field
             Preconditions.checkArgument(
                 schemaField == null || schemaField.fieldId() == sourceColumnId,
                 "Cannot create identity partition sourced from different field in schema: %s",
                 name);
   ```
   
   Let me check if we should compare `fieldId` with each `column id`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2886096747

   Hey guys. I was traveling this week. I'm now back on this pr, updating according to the comments. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3195471564

   I'm resuming the work on this one (I have on vacation for the last two weeks).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3434635838

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3172256678

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061143136


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecParser.java:
##########
@@ -95,6 +96,56 @@ public void testFromJsonWithFieldId() {
     assertThat(spec.fields().get(1).fieldId()).isEqualTo(1000);
   }
 
+  @TestTemplate
+  public void testFromJsonWithSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"source-ids\" : [ 1, 2 ],\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"source-ids\" : [ 2, 3 ],\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    PartitionSpec spec = PartitionSpecParser.fromJson(table.schema(), specString);
+
+    assertThat(spec.fields()).hasSize(2);
+    assertThat(spec.fields().get(0).sourceIds()).hasSize(2);
+    assertThat(spec.fields().get(1).sourceIds()).hasSize(2);
+  }
+
+  @TestTemplate
+  public void testFromJsonWithoutSourceIdAndSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    assertThatThrownBy(
+            () -> {
+              PartitionSpecParser.fromJson(table.schema(), specString);
+            })
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage(
+            "Cannot parse partition field, either source-id or source-ids has to be present");
+  }

Review Comment:
   I added a test in `TestPartitionSpecParser` testing when neither `source-id` and `source-ids` are provided: `testFromJsonWithoutSourceIdAndSourceIds()`.
   I will add additional tests regarding your previous comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061155133


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +147,29 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
+      Preconditions.checkArgument(
+          element.has(SOURCE_ID) || element.has(SOURCE_IDS),
+          "Cannot parse partition field, either "
+              + SOURCE_ID
+              + " or "
+              + SOURCE_IDS
+              + " has to be present");

Review Comment:
   I added a check to guarantee we only have `source-id` or `source-ids`, not both (and I added a corresponding test for that).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060354754


##########
.palantir/revapi.yml:
##########
@@ -1177,6 +1177,10 @@ acceptedBreaks:
       old: "class org.apache.iceberg.Metrics"
       new: "class org.apache.iceberg.Metrics"
       justification: "Java serialization across versions is not guaranteed"
+    - code: "java.class.defaultSerializationChanged"

Review Comment:
   Yes, We don't use it. But I don't see a general guidelines about backward compatibility for Java serialization. Let us wait for others opinion. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060454170


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated

Review Comment:
   That's a good point. The reason why I flagged deprecated is to "force" the callers to use source-ids "style" (I don't deprecated in the builder as the current transforms support only one argument).
   I'm OK to remove the `@Deprecated` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2095563010


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -627,7 +636,10 @@ PartitionSpec buildUnchecked() {
   static void checkCompatibility(PartitionSpec spec, Schema schema) {
     final Map<Integer, Integer> parents = TypeUtil.indexParents(schema.asStruct());
     for (PartitionField field : spec.fields) {
-      Type sourceType = schema.findType(field.sourceId());
+      List<Type> sourceTypes = Lists.newArrayList();
+      for (int sourceId : field.sourceIds()) {
+        sourceTypes.add(schema.findType(sourceId));
+      }

Review Comment:
   nit: newline



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2093530131


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -118,7 +128,12 @@ public String transformAsString() {
     }
 
     public int sourceId() {
-      return sourceId;
+      Preconditions.checkArgument(sourceIds.size() == 1, "Only one source is expected");

Review Comment:
   I updated the precondition to support multi-args.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2905590955

   Finally back from several trips, so resuming work on this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3310067611

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3596875523

   Back again :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2095568633


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -88,14 +89,23 @@ Builder withSpecId(int newSpecId) {
       return this;
     }
 
+    Builder addField(
+        String transformAsString, List<Integer> sourceIds, int partitionId, String name) {

Review Comment:
   null checks?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2105295782


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -602,17 +602,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {

Review Comment:
   Practically, sourceIds element can't be null. But happy to add a check πŸ˜„ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2105296456


##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -88,14 +89,23 @@ Builder withSpecId(int newSpecId) {
       return this;
     }
 
+    Builder addField(
+        String transformAsString, List<Integer> sourceIds, int partitionId, String name) {

Review Comment:
   Same comment: let me add a null element check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060457399


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated
   PartitionField(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-    this.sourceId = sourceId;
+    this(List.of(sourceId), fieldId, name, transform);
+  }
+
+  PartitionField(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+    this.sourceIds = sourceIds;
     this.fieldId = fieldId;
     this.name = name;
     this.transform = transform;
   }
 
-  /** Returns the field id of the source field in the {@link PartitionSpec spec's} table schema. */
+  /**
+   * Returns the source field in the {@link PartitionSpec spec's} table schema.
+   *
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use sourceIds() instead.
+   */
+  @Deprecated
   public int sourceId() {
-    return sourceId;
+    Preconditions.checkArgument(sourceIds.size() == 1, "Only one source is supported");

Review Comment:
   Message here is a little confusing maybe something like  `Cannot use sourceId() on a transform with multiple arguments"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2064268775


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -625,7 +634,7 @@ PartitionSpec buildUnchecked() {
 
   static void checkCompatibility(PartitionSpec spec, Schema schema) {
     for (PartitionField field : spec.fields) {
-      Type sourceType = schema.findType(field.sourceId());
+      Type sourceType = schema.findType(field.sourceIds().get(0));

Review Comment:
   We should be extending this logic as well to handle multi arg sources



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060518830


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecParser.java:
##########
@@ -95,6 +96,56 @@ public void testFromJsonWithFieldId() {
     assertThat(spec.fields().get(1).fieldId()).isEqualTo(1000);
   }
 
+  @TestTemplate
+  public void testFromJsonWithSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"source-ids\" : [ 1, 2 ],\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"source-ids\" : [ 2, 3 ],\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    PartitionSpec spec = PartitionSpecParser.fromJson(table.schema(), specString);
+
+    assertThat(spec.fields()).hasSize(2);
+    assertThat(spec.fields().get(0).sourceIds()).hasSize(2);
+    assertThat(spec.fields().get(1).sourceIds()).hasSize(2);
+  }
+
+  @TestTemplate
+  public void testFromJsonWithoutSourceIdAndSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    assertThatThrownBy(
+            () -> {
+              PartitionSpecParser.fromJson(table.schema(), specString);
+            })
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage(
+            "Cannot parse partition field, either source-id or source-ids has to be present");
+  }

Review Comment:
   We should have some tests for "toJson" as well? 
   
   We also need a check that the validation for identity transforms still holds true. Ie you cannot make a multi-arg transform with the same name as any of the columns



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061143136


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecParser.java:
##########
@@ -95,6 +96,56 @@ public void testFromJsonWithFieldId() {
     assertThat(spec.fields().get(1).fieldId()).isEqualTo(1000);
   }
 
+  @TestTemplate
+  public void testFromJsonWithSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"source-ids\" : [ 1, 2 ],\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"source-ids\" : [ 2, 3 ],\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    PartitionSpec spec = PartitionSpecParser.fromJson(table.schema(), specString);
+
+    assertThat(spec.fields()).hasSize(2);
+    assertThat(spec.fields().get(0).sourceIds()).hasSize(2);
+    assertThat(spec.fields().get(1).sourceIds()).hasSize(2);
+  }
+
+  @TestTemplate
+  public void testFromJsonWithoutSourceIdAndSourceIds() {
+    String specString =
+        "{\n"
+            + "  \"spec-id\" : 1,\n"
+            + "  \"fields\" : [ {\n"
+            + "    \"name\" : \"id_bucket\",\n"
+            + "    \"transform\" : \"bucket[8]\",\n"
+            + "    \"field-id\" : 1001\n"
+            + "  }, {\n"
+            + "    \"name\" : \"data_bucket\",\n"
+            + "    \"transform\" : \"bucket[16]\",\n"
+            + "    \"field-id\" : 1000\n"
+            + "  } ]\n"
+            + "}";
+
+    assertThatThrownBy(
+            () -> {
+              PartitionSpecParser.fromJson(table.schema(), specString);
+            })
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage(
+            "Cannot parse partition field, either source-id or source-ids has to be present");
+  }

Review Comment:
   I added a test in `TestPartitionSpecParser` testing when neither `source-id` and `source-ids` are provided: `testFromJsonWithoutSourceIdAndSourceIds()`.
   I will add additional tests for `toJson` path too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060214278


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -601,17 +601,26 @@ public Builder alwaysNull(String sourceName) {
 
     // add a partition field with an auto-increment partition field id starting from
     // PARTITION_DATA_ID_START
+    Builder add(List<Integer> sourceIds, String name, Transform<?, ?> transform) {
+      return add(sourceIds, nextFieldId(), name, transform);
+    }
+
     Builder add(int sourceId, String name, Transform<?, ?> transform) {
-      return add(sourceId, nextFieldId(), name, transform);
+      return add(List.of(sourceId), name, transform);
     }
 
-    Builder add(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-      checkAndAddPartitionName(name, sourceId);
-      fields.add(new PartitionField(sourceId, fieldId, name, transform));
+    Builder add(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+      // we use the first entry in the source-ids list here
+      checkAndAddPartitionName(name, sourceIds.get(0));

Review Comment:
   For now, I'm using the first element. I can update `checkAndAddPartitionName` to consider all elements in source ids.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060216106


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -98,7 +101,16 @@ static void toJsonFields(UnboundPartitionSpec spec, JsonGenerator generator) thr
       generator.writeStartObject();
       generator.writeStringField(NAME, field.name());
       generator.writeStringField(TRANSFORM, field.transformAsString());
-      generator.writeNumberField(SOURCE_ID, field.sourceId());
+      if (field.sourceIds().size() == 1) {
+        generator.writeNumberField(SOURCE_ID, field.sourceId());
+      } else {
+        generator.writeFieldName(SOURCE_IDS);
+        generator.writeStartArray();
+        for (Integer value : field.sourceIds()) {
+          generator.writeNumber(value);
+        }
+        generator.writeEndArray();
+      }

Review Comment:
   Ack, fixing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2060449197


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated

Review Comment:
   I don't actually have a problem with keeping this method? It's essentially just going to save callers doing a List.of(element) and I don't think we are are really adding much complexity?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2061154925


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated

Review Comment:
   I removed the deprecation.



##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,49 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
+  /**
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use PartitionField(sourceIds, fieldId,
+   *     name, transform) instead.
+   */
+  @Deprecated
   PartitionField(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-    this.sourceId = sourceId;
+    this(List.of(sourceId), fieldId, name, transform);
+  }
+
+  PartitionField(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+    this.sourceIds = sourceIds;
     this.fieldId = fieldId;
     this.name = name;
     this.transform = transform;
   }
 
-  /** Returns the field id of the source field in the {@link PartitionSpec spec's} table schema. */
+  /**
+   * Returns the source field in the {@link PartitionSpec spec's} table schema.
+   *
+   * @deprecated since 1.10.0, will be removed in 2.0.0; use sourceIds() instead.
+   */
+  @Deprecated
   public int sourceId() {
-    return sourceId;
+    Preconditions.checkArgument(sourceIds.size() == 1, "Only one source is supported");

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3146046422

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3949268774

   I'm back (again πŸ˜„ ) on Iceberg, resuming my different PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2908586432


##########
core/src/main/java/org/apache/iceberg/PartitionSpecParser.java:
##########
@@ -132,15 +144,37 @@ private static void buildFromJsonFields(UnboundPartitionSpec.Builder builder, Js
 
       String name = JsonUtil.getString(NAME, element);
       String transform = JsonUtil.getString(TRANSFORM, element);
-      int sourceId = JsonUtil.getInt(SOURCE_ID, element);
 
-      // partition field ids are missing in old PartitionSpec, they always auto-increment from
-      // PARTITION_DATA_ID_START
-      if (element.has(FIELD_ID)) {
-        builder.addField(transform, sourceId, JsonUtil.getInt(FIELD_ID, element), name);
-        fieldIdCount++;
+      if (element.has(SOURCE_IDS)) {

Review Comment:
   I think we should at least warn here if source id is also present and has a different value. I could also see throwing an exception but we generally don't fail unless we absolutely have to when parsing. As long as the writer would override the bad source-id I think we are ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2908575226


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########


Review Comment:
   Since it indexes by the first sourceId i'm not sure what will actually happen, we probably need to do something here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#discussion_r2931251311


##########
api/src/main/java/org/apache/iceberg/PartitionField.java:
##########
@@ -19,26 +19,42 @@
 package org.apache.iceberg;
 
 import java.io.Serializable;
+import java.util.List;
 import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
 import org.apache.iceberg.transforms.Transform;
 
 /** Represents a single field in a {@link PartitionSpec}. */
 public class PartitionField implements Serializable {
-  private final int sourceId;
+  private final List<Integer> sourceIds;
   private final int fieldId;
   private final String name;
   private final Transform<?, ?> transform;
 
   PartitionField(int sourceId, int fieldId, String name, Transform<?, ?> transform) {
-    this.sourceId = sourceId;
+    this(ImmutableList.of(sourceId), fieldId, name, transform);
+  }
+
+  PartitionField(List<Integer> sourceIds, int fieldId, String name, Transform<?, ?> transform) {
+    this.sourceIds = ImmutableList.copyOf(sourceIds);
     this.fieldId = fieldId;
     this.name = name;
     this.transform = transform;
   }
 
   /** Returns the field id of the source field in the {@link PartitionSpec spec's} table schema. */
   public int sourceId() {
-    return sourceId;
+    return sourceIds.get(0);

Review Comment:
   I added an accessor where I added a `Preconditions.checkState()`:
   
   ```
     public int sourceId() {
         Preconditions.checkState(
             sourceIds.size() == 1,
             "Use sourceIds() for multi-argument transforms, found %s source ids",
             sourceIds.size());
         return sourceIds.get(0);
     }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3318554037

   I just did a quick "rebase quick fix". I'm resuming my work on this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #12897:
URL: https://github.com/apache/iceberg/pull/12897#issuecomment-3479861793

   Here we go again πŸ˜„ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org