You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sl255051 (via GitHub)" <gi...@apache.org> on 2024/07/10 16:25:43 UTC

[PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

sl255051 opened a new pull request, #10678:
URL: https://github.com/apache/iceberg/pull/10678

   Make PartitionSpec.Builder search for columns in a case-insensitive way, i.e. `"COLUMN_1" == "column_1"`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696029266


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -95,6 +95,19 @@ public void testSpecInfoPartitionedTable() {
         .doesNotContainKey(Integer.MAX_VALUE);
   }
 
+  @TestTemplate
+  public void testSpecInfoPartitionedTableCaseInsensitive() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(schema).caseSensitive(false).identity("DATA").build();

Review Comment:
   test case has been added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699225863


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   The current behavior is to fail when a schema that requires case sensitivity is used in a case insensitive way. Right now that would be by throwing an exception when [creating the case insensitive index](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/types/TypeUtil.java#L184-L190).
   
   Because Iceberg needs to work in contexts that are both case sensitive and case insensitive, I think it is the right strategy to fail at runtime. Iceberg can't impose case insensitivity on engines that are case sensitive or vice versa.
   
   In nearly all situations, this works fine and is not noticed because it is very uncommon to have both `data` and `DATA` in a schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700962527


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   Yes, I am sure I'm getting `RegularImmutableMap`
   
   <img width="848" alt="image" src="https://github.com/user-attachments/assets/38841220-bcb2-4eab-880f-4994038dc7ab">
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707675526


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);

Review Comment:
   I think this check could be much shorter:
   
   ```java
   String key = name.toLowerCase(Locale.ROOT);
   Integer existingId = indexByLowerCaseName.put(key);
   Preconditions.checkArgument(
       existingId == null || existingId = fieldId,
       "Cannot build lower case index: %s and %s collide",
       byId.get(fieldId),
       byId.get(existingId));
   ```
   
   Note that I'm using the `byId` index referenced in the comment below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702369242


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -182,11 +181,10 @@ public static Map<Integer, String> indexQuotedNameById(
   }
 
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
-    Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
+    ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
     indexByName(struct)
-        .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
-    return indexByLowerCaseName;
+        .forEach((name, integer) -> builder.put(name.toLowerCase(Locale.ROOT), integer));
+    return builder.buildOrThrow();

Review Comment:
   Instead of depending on ImmutableMap builder for the dup check, I am wondering if we should handle the collision check here in order to throw an exception with more informative error msg than `IllegalArgumentException: Multiple entries with same key: a=2 and a=1`. E.g., the error msg can be sth like `Unable to build case insensitive mapping from field name to field id because two fields have the same lower case name...`. We can also get the original field name since we know the two field ids and have the original schema/struct.
   
   also let's add Javadoc to this method to document the behavior under collision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700986212


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   I updated the code to use `ImmutableMap.Builder`. It will now throw on duplicate keys. It still returns a RegularImmutableMap instance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707681345


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);
+              if (prevValue != null) {
+                Types.NestedField field1 = struct.field(prevValue);
+                Types.NestedField field2 = struct.field(integer);

Review Comment:
   In a struct, `field` returns a child field. But the index this `forEach` is traversing is the index of all fields, so there will be IDs that are not returned by `field`, resulting in a `NullPointerException` when formatting the exception message.
   
   The fix is to use `IndexByName` directly and get both the `byName()` and `byId()` maps:
   
   ```java
     IndexByName indexer = new IndexByName();
     visit(struct, indexer);
     Map<String, Integer> byName = indexer.byName();
     Map<Integer, String> byId = indexer.byId();
     ...
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sfc-gh-rspitzer (via GitHub)" <gi...@apache.org>.
sfc-gh-rspitzer commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257232552

   In light of @rdblue 's comment let's make sure we add a test case which has a partition spec with
   "both foo and Foo" as different but referenced source columns. 
   
   CREATE TABLE <foo int, Foo int> PARTITIONED BY (foo, Foo)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) {
     }
 
     private void checkAndAddPartitionName(String name, Integer sourceColumnId) {
-      Types.NestedField schemaField = schema.findField(name);
+      Types.NestedField schemaField =

Review Comment:
   I already investigated this and the answer is I don't believe so. `findSourceColumn` will throw an exception if the column isn't found whereas `checkAndAddPartitionName` can still succeed even if the column isn't found.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707675526


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);

Review Comment:
   I think this check could be much shorter:
   
   ```java
   String key = name.toLowerCase(Locale.ROOT);
   Integer existingId = indexByLowerCaseName.put(key);
   Preconditions.checkArgument(
       existingId == null || existingId = fieldId,
       "Cannot build lower case index: %s and %s collide",
       idToName.get(fieldId),
       idToName.get(existingId));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707740448


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);
+              if (prevValue != null) {
+                Types.NestedField field1 = struct.field(prevValue);
+                Types.NestedField field2 = struct.field(integer);
+                throw new IllegalArgumentException(
+                    String.format(
+                        "Unable to build field name to id mapping because two fields have the same lower case name: %s and %s",

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707732421


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name

Review Comment:
   I respectfully disagree that IllegalStateException is more accurate. To me "state" is preexists a method call. State applies more to an instance method. In this case this is a static method; there is no class and there is no state. The argument  `struct` is the only input to this method and that `struct` is invalid for indexing by lower case name.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699234156


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @rdblue The timing of your answer is impeccable because at the very moment I received your response I was debugging that very method, `TypeUtil.indexByLowerCaseName()`, to see how it handles `data` and `DATA`. The answer is that the method does not throw an exception at all.
   
   If you start with this schema:
   ```
   Schema schema = new Schema(
           required(1, "id", Types.IntegerType.get()),
           required(2, "data", Types.StringType.get()),
           required(3, "DATA", Types.StringType.get()));
   ```
   then calling `TypeUtil.indexByLowerCaseName()` will return a RegularImmutableMap of
   ```
   {
       "data": 3,
       "id": 1
   }
   ```
   So either the result is undefined or (by implementation detail) is the ID of the last column with case-insensitive matching name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1701030165


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   I don't think this should normalize the target name. That could be passed by the caller and this should not modify it. I think it should continue to use the target name that was passed in.
   
   The problem isn't with this method, it is with this version of the method, it is with the `identity(String sourceName)` version that assumes it can pass the source name as the target name. One solution is to lower case the string there, but I would prefer to preserve the case from the schema instead.
   
   I think the easiest way to do that is to add a private identity method, like this:
   
   ```java
       Builder identity(String sourceName, String targetName) {
         return identity(findSourceColumn(sourceName), targetName);
       }
   
       Builder identity(String sourceName) {
         Types.NestedField sourceColumn = findSourceColumn(sourceName);
         return identity(sourceColumn, sourceColumn.name);
       }
   
       private Builder identity(String sourceColumn, String targetName) {
         checkAndAddPartitionName(targetName, sourceColumn.fieldId());
         PartitionField field =
             new PartitionField(
                 sourceColumn.fieldId(), nextFieldId(), targetName, Transforms.identity());
         checkForRedundantPartitions(field);
         fields.add(field);
         return this;
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696126712


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   Now you're pushing me into the deep end of the pool. 😉 I can't answer your question. Your question would have to be directed to someone with greater knowledge of Iceberg's case-insensitive plan.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700713501


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   Oh, I see what I missed. It isn't using `ImmutableMap.Builder`. It is creating the lower case index with `Maps.newHashMap()`. I just read it wrong yesterday. Are you sure you're getting a `RegularImmutableMap`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698118


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);

Review Comment:
   `Preconditions.checkArgument` throws IllegalArgumentException. This conflicts with your other comment that suggests throwing IllegalStateException. What's the path forward?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698118


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);

Review Comment:
   I've updated the code to use `Preconditions.checkArgument`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2264183829

   > > In light of @rdblue 's comment let's make sure we add a test case which has a partition spec with "both foo and Foo" as different but referenced source columns.
   > > CREATE TABLE <foo int, Foo int> PARTITIONED BY (foo, Foo)
   > 
   > @sfc-gh-rspitzer where would I add this test?
   
   I've added several unit tests that address the scenario. (I used columns named `data` and `DATA` rather than `foo` and `Foo`)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699234156


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @rdblue The timing of your answer is impeccable because at the very moment I received your answer I was debugging that very method, `TypeUtil.indexByLowerCaseName()`, to see how it handles `data` and `DATA`. The answer is that the method does not throw an exception at all.
   
   If you start with this schema:
   ```
   Schema schema = new Schema(
           required(1, "id", Types.IntegerType.get()),
           required(2, "data", Types.StringType.get()),
           required(3, "DATA", Types.StringType.get()));
   ```
   then calling `TypeUtil.indexByLowerCaseName(schema.struct)` will return a RegularImmutableMap of
   ```
   {
       "data": 3,
       "id": 1
   }
   ```
   So either the result is undefined or (by implementation detail) is the ID of the last column with case-insensitive matching name. But the bottom line is there is no runtime failure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1723721641


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "partition1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "partition1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);

Review Comment:
   @sl255051, what's the purpose of this test? It just tests the default behavior with one field and no conflicts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2253353743

   I've updated the PR with a case-sensitivity flag pattern I've found in other parts of the Iceberg codebase. I hope this change is more paletable to the Iceberg community.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700962527


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   Yes, I double-checked it to be sure I'm getting `RegularImmutableMap`
   
   <img width="848" alt="image" src="https://github.com/user-attachments/assets/38841220-bcb2-4eab-880f-4994038dc7ab">
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707690572


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -20,6 +20,7 @@
 
 import static org.apache.iceberg.types.Types.NestedField.required;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;

Review Comment:
   @nastra I thought that we specifically did not use static imports. Is it okay for assertions now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707696368


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =

Review Comment:
   I don't think this is correct. Schemas are neither case sensitive or case insensitive. They can be used in a case sensitive or insensitive context.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707739853


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =

Review Comment:
   These constants have been renamed to SCHEMA_WITHOUT_NAME_CONFLICTS and SCHEMA_WITH_NAME_CONFLICTS



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707743014


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);
+              if (prevValue != null) {
+                Types.NestedField field1 = struct.field(prevValue);
+                Types.NestedField field2 = struct.field(integer);

Review Comment:
   I made the change but I don't see where I would need to use `byName` because this method doesn't currently lookup any field by name. All field lookups are currently done by ID.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2225993288

   > @sl255051 appreciate you are taking the stub for the PR.
   > 
   > But I am wondering why do you think column name case insensitivity is the right behavior when building PartitionSpec? I think in iceberg schema we can have both column named `data` and `DATA` with each different field id assigned, like below
   > 
   > ```
   > table {
   >   1: id: required int
   >   2: data: required string
   >   3: DATA: required string
   > }
   > ```
   > 
   > Would this change introduce additional ambiguity when resolve a column name in a case insensitive way?
   
   Thanks for taking the time to review my PR. I did notice that the Schema object uses a simple Map<String, Integer> for column names which means the schema is case sensitive. But I wonder if that is a bug too. I believe partition columns should be case-insensitive based on this issue https://github.com/apache/iceberg/issues/83. That issue says to make Iceberg case-insensitive. I can see lots of work was done to enable case-insensitivity in Iceberg. Several objects even have multiple methods to enable case-insensitivity. Take the Schema object as an example. If case-insensitivity is not a feature of Iceberg why would that class have both methods, `findField` and `caseInsensitiveFindField`?
   
   In summary, I believe case-insensitivity is the correct path forward. I can accept that I may not have implemented in the best way. If that is the case I would appreciate some pointers on how best to implement case-insensitivity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702992594


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -182,11 +181,10 @@ public static Map<Integer, String> indexQuotedNameById(
   }
 
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
-    Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
+    ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
     indexByName(struct)
-        .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
-    return indexByLowerCaseName;
+        .forEach((name, integer) -> builder.put(name.toLowerCase(Locale.ROOT), integer));
+    return builder.buildOrThrow();

Review Comment:
   I've made the change you proposed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702998105


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   I've reverted normalization of `targetName` in those methods too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257241120

   @rdblue I am not certain I understand your suggested path forward. Are you suggesting some edit like this at the bottom of PartitionSpec.Builder.checkAndAddPartitionName?
   
   ```
         String partitionName = schemaField != null ? schemaField.name() : name;
         Preconditions.checkArgument(!partitionName.isEmpty(), "Cannot use empty partition name: %s", name);
         Preconditions.checkArgument(
             !partitionNames.contains(partitionName), "Cannot use partition name more than once: %s", name);
         partitionNames.add(partitionName);
       }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257241580

   > In light of @rdblue 's comment let's make sure we add a test case which has a partition spec with "both foo and Foo" as different but referenced source columns.
   > 
   > CREATE TABLE <foo int, Foo int> PARTITIONED BY (foo, Foo)
   
   @sfc-gh-rspitzer where would I add this test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #10678:
URL: https://github.com/apache/iceberg/pull/10678


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702977668


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,697 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "p1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "p1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketSourceNameDoesNotAllowExactDuplicateWhenCaseSensitive() {
+    assertThatIllegalArgumentException()
+        .isThrownBy(
+            () ->
+                PartitionSpec.builderFor(SCHEMA_CASE_SENSITIVE)
+                    .bucket("data", 10, "p1")

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699234156


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @rdblue The timing of your answer is impeccable because at the very moment I received your answer I was debugging that very method, `TypeUtil.indexByLowerCaseName()`, to see how it handles `data` and `DATA`. The answer is that the method does not throw an exception at all.
   
   If you start with this schema:
   ```
   Schema schema = new Schema(
           required(1, "id", Types.IntegerType.get()),
           required(2, "data", Types.StringType.get()),
           required(3, "DATA", Types.StringType.get()));
   ```
   then calling `TypeUtil.indexByLowerCaseName(schema)` will return a RegularImmutableMap of
   ```
   {
       "data": 3,
       "id": 1
   }
   ```
   So either the result is undefined or (by implementation detail) is the ID of the last column with case-insensitive matching name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700711773


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   I thought that `ImmutableMap.Builder` would reject duplicate keys. In any case, we should probably update the code I linked to so that it catches this and throws a nice error message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707732421


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name

Review Comment:
   I, too, think IllegalStateException is more semantically correct, but this suggestion conflicts with your other suggestion to use Preconditions.checkArgument which throws IllegalArgumentException. Which solution has higher precedence, throwing IllegalStateException or calling Preconditions.checkArgument?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1701221811


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   Are you implying that the normalization of `targetName` in the other transform methods is OK to keep?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707662722


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);
+              if (prevValue != null) {
+                Types.NestedField field1 = struct.field(prevValue);
+                Types.NestedField field2 = struct.field(integer);
+                throw new IllegalArgumentException(
+                    String.format(
+                        "Unable to build field name to id mapping because two fields have the same lower case name: %s and %s",

Review Comment:
   We typically use "Cannot" rather than "Unable to". Also, this could be more direct.
   
   How about "Cannot build lower case index: %s and %s collide"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707689251


##########
api/src/test/java/org/apache/iceberg/TestSchemaCaseSensitivity.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestSchemaCaseSensitivity {
+
+  @Test
+  public void
+      testSchemaWithColumnNamesThatDifferOnlyInLetterCaseThrowsOnCaseInsensitiveFindField() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.LongType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()));
+    assertThatIllegalArgumentException()
+        .isThrownBy(() -> schema.caseInsensitiveFindField("DATA"))
+        .withMessage(
+            "Unable to build field name to id mapping because two fields have the same lower case name: data and DATA");
+  }
+
+  @Test
+  public void
+      testSchemaWithColumnNamesThatDifferOnlyInLetterCaseSucceedsOnCaseSensitiveFindField() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.LongType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()));
+
+    Types.NestedField actual1 = schema.findField("data");
+    assertThat(actual1).isEqualTo(Types.NestedField.required(2, "data", Types.StringType.get()));
+    Types.NestedField actual2 = schema.findField("DATA");
+    assertThat(actual2).isEqualTo(Types.NestedField.required(3, "DATA", Types.StringType.get()));
+  }
+
+  @Test
+  public void testCaseInsensitiveFindFieldSucceeds() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.LongType.get()), required(2, "data", Types.StringType.get()));
+
+    Types.NestedField actual1 = schema.caseInsensitiveFindField("Data");

Review Comment:
   Looking for `DATA` (all caps) would fit with the other test cases better and make it more obvious that this is where the case differs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2272052551

   @sl255051, I think this is the right fix for the test failures from the changes to the identity transform method:
   
   ```diff
   diff --git a/api/src/main/java/org/apache/iceberg/PartitionSpec.java b/api/src/main/java/org/apache/iceberg/PartitionSpec.java
   index 6f5eb70297..a11c2b8537 100644
   --- a/api/src/main/java/org/apache/iceberg/PartitionSpec.java
   +++ b/api/src/main/java/org/apache/iceberg/PartitionSpec.java
   @@ -465,7 +465,7 @@ public class PartitionSpec implements Serializable {
    
        public Builder identity(String sourceName) {
          Types.NestedField sourceColumn = findSourceColumn(sourceName);
   -      return identity(sourceColumn, sourceColumn.name());
   +      return identity(sourceColumn, schema.findColumnName(sourceColumn.fieldId()));
        }
    
        public Builder year(String sourceName, String targetName) {
   ```
   
   The problem was that in the deletes metadata table, the schema is this:
   
   ```
   table {
     2147483546: file_path: required string (Path of a file in which a deleted row is stored)
     2147483545: pos: required long (Ordinal position of a deleted row in the data file)
     2147483544: row: optional struct<1: c1: optional int, 2: c2: optional string, 3: c3: optional string> (Deleted row values)
     2147483642: partition: required struct<4: c1: optional int> (Partition that position delete row belongs to)
     2147483643: spec_id: required int (Spec ID used to track the file containing a row)
   }
   ```
   
   There are two `c1` columns and now that this uses `sourceColumn.name()` there was a duplicate `c1`. Before this, the name used to look up the nested field would have been used. That is, both `row.c1` and `partition.c1` so there was no conflict.
   
   Using the full name of the field for the partition name fixes the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698954


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);

Review Comment:
   I don't think that this test case needs to create a table and write content to disk. This can test the partition spec's partition type instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707707143


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "partition1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "partition1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketSourceNameAllowsExactDuplicateWhenCaseSensitive() {
+    Schema schema = SCHEMA_CASE_SENSITIVE;

Review Comment:
   Why is this introducing a local variable for the constant?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2226008338

   > > @sl255051 appreciate you are taking the stub for the PR.
   > > But I am wondering why do you think column name case insensitivity is the right behavior when building PartitionSpec? I think in iceberg schema we can have both column named `data` and `DATA` with each different field id assigned, like below
   > > ```
   > > table {
   > >   1: id: required int
   > >   2: data: required string
   > >   3: DATA: required string
   > > }
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > Would this change introduce additional ambiguity when resolve a column name in a case insensitive way?
   > 
   > Thanks for taking the time to review my PR. I did notice that the Schema object uses a simple Map<String, Integer> for column names which means the schema is case sensitive. But I wonder if that is a bug too. I believe partition columns should be case-insensitive based on this issue #83. That issue says to make Iceberg case-insensitive. I can see lots of work was done to enable case-insensitivity in Iceberg. Several objects even have multiple methods to enable case-insensitivity. Take the Schema object as an example. If case-insensitivity is not a feature of Iceberg why would that class have both methods, `findField` and `caseInsensitiveFindField`?
   > 
   > In summary, I believe case-insensitivity is the correct path forward. I can accept that I may not have implemented in the best way. If that is the case I would appreciate some pointers on how best to implement case-insensitivity.
   
   I am not fully aware of the current status of case sensitivity support in iceberg as it's not documented in the spec, maybe we can ask if any of the experts want to chime in  @rdblue or @RussellSpitzer
   
   But as you mentioned if current schema supports case sensitivity, I dont think it's correct to build partition spec when finding column by name in a case insensitive manner, as it introduce additional ambiguity per my example illustrated above.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696038786


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;

Review Comment:
   Can we just have a void `caseInsensitiveFields` API? The default is false (preserving the original behavior), and then when this API gets called it sets it to true.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696192362


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   the question is really meant for @rdblue and @RussellSpitzer :) There is also the question of engine interoperability on case sensitivity. If the table is defined as case insensitive, ideally validation should reject the schema that @dramaticlly pointed out.
   
   ```
   table {
     1: id: required int
     2: data: required string
     3: DATA: required string
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1709778268


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -20,6 +20,7 @@
 
 import static org.apache.iceberg.types.Types.NestedField.required;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;

Review Comment:
   during the JUni4 -> JUnit5 migration we used static imports of `assertThat` for better readability at places that didn't use AssertJ assertions before and switched from JUnit4 asserts to AssertJ asserts. At that point the codebase was split between static import usage of `assertThat` and non-static usage and so we eventually enforced static imports for AssertJ assertions via https://github.com/apache/iceberg/pull/10517



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707699728


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =

Review Comment:
   Instead, how about with or without name conflicts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707743014


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name
+   */
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
     Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
     indexByName(struct)
         .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
+            (name, integer) -> {
+              String key = name.toLowerCase(Locale.ROOT);
+              Integer prevValue = indexByLowerCaseName.get(key);
+              if (prevValue != null) {
+                Types.NestedField field1 = struct.field(prevValue);
+                Types.NestedField field2 = struct.field(integer);

Review Comment:
   I made the suggested change but without this line
   ```
   Map<String, Integer> byName = indexer.byName();
   ```
   I don't see where I would need to use `byName` because this method doesn't currently lookup any field by name. All field lookups are currently done by ID.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707750094


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -20,6 +20,7 @@
 
 import static org.apache.iceberg.types.Types.NestedField.required;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;

Review Comment:
   static imports for assertions became allowed by this commit to checkstyle.xml
   https://github.com/apache/iceberg/commit/b3261d07fc687d5df1f0f8309bc5a0f6022434cf



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2273922553

   > @sl255051, I think this is the right fix for the test failures from the changes to the identity transform method:
   
   @rdblue I've implemented your suggested change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1701214301


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   I assume you meant this. The two salient differences are 
   
   1. adding `public` to `Builder identity(String sourceName)`
   2. changing `sourceColumn` type from `String` to `Types.NestedField` in `private Builder identity(String sourceColumn, String targetName))`
   
   
   ```
       Builder identity(String sourceName, String targetName) {
         return identity(findSourceColumn(sourceName), targetName);
       }
   
       public Builder identity(String sourceName) {
         Types.NestedField sourceColumn = findSourceColumn(sourceName);
         return identity(sourceColumn, sourceColumn.name());
       }
   
       private Builder identity(Types.NestedField sourceColumn, String targetName) {
         checkAndAddPartitionName(targetName, sourceColumn.fieldId());
         PartitionField field =
                 new PartitionField(
                         sourceColumn.fieldId(), nextFieldId(), targetName, Transforms.identity());
         checkForRedundantPartitions(field);
         fields.add(field);
         return this;
       }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1701214301


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   I assume you meant this. The two salient differences are 
   
   1. adding `public` to `Builder identity(String sourceName)`
   2. changing `sourceColumn` type from `String` to `Types.NestedField` in `private Builder identity(String sourceColumn, String targetName))`
   
   
   ```Java
       Builder identity(String sourceName, String targetName) {
         return identity(findSourceColumn(sourceName), targetName);
       }
   
       public Builder identity(String sourceName) {
         Types.NestedField sourceColumn = findSourceColumn(sourceName);
         return identity(sourceColumn, sourceColumn.name());
       }
   
       private Builder identity(Types.NestedField sourceColumn, String targetName) {
         checkAndAddPartitionName(targetName, sourceColumn.fieldId());
         PartitionField field =
                 new PartitionField(
                         sourceColumn.fieldId(), nextFieldId(), targetName, Transforms.identity());
         checkForRedundantPartitions(field);
         fields.add(field);
         return this;
       }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699209144


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @stevenzwu @rdblue @RussellSpitzer I suppose the answer to this question is to NOT use a case-sensitive schema in a case-insensitive way. In other words, if the schema is case-sensitive and contains columns named Make and MAKE then the result of calling `caseInsensitiveFindField("Make")` is undefined.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699210022


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;

Review Comment:
   > current `caseSensitive(boolean)` seems more consistent with existing code, like `Scan` interface.
   
   That is exactly why I implemented it this way



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699209144


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @stevenzwu @rdblue @RussellSpitzer I suppose the answer to this question is to NOT use a case-sensitive schema in a case-insensitive way. In other words, if the schema is case-sensitive and contains columns named `Make` and `MAKE` then the result of calling `caseInsensitiveFindField("Make")` is undefined.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707706083


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "partition1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "partition1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);

Review Comment:
   I'm not sure that this is testing anything relevant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707740896


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -450,13 +459,21 @@ Builder identity(String sourceName, String targetName) {
       return this;
     }
 
+    Builder identity(String sourceName, String targetName) {
+      return identity(findSourceColumn(sourceName), targetName);
+    }

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707750094


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -20,6 +20,7 @@
 
 import static org.apache.iceberg.types.Types.NestedField.required;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;

Review Comment:
   static imports for assertions became allowed by this commit
   https://github.com/apache/iceberg/commit/b3261d07fc687d5df1f0f8309bc5a0f6022434cf



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707763657


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "partition1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "partition1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketSourceNameAllowsExactDuplicateWhenCaseSensitive() {
+    Schema schema = SCHEMA_CASE_SENSITIVE;

Review Comment:
   because the schema is referenced twice in this method, line 110 and line 121. I decided to use a variable because  while writing these test variations I had some copy/paste bugs where, for example, line 110 could have been referencing SCHEMA_CASE_SENSITIVE whereas line 121 could have been referencing SCHEMA_CASE_INSENSITIVE. Using a variable makes sure the entire test method is using the same schema.
   
   But I don't mind removing the local variable if that makes the code more consistent with Iceberg coding preference / precedence



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707655607


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -450,13 +459,21 @@ Builder identity(String sourceName, String targetName) {
       return this;
     }
 
+    Builder identity(String sourceName, String targetName) {
+      return identity(findSourceColumn(sourceName), targetName);
+    }

Review Comment:
   Nit: for the other cases, the methods are in a different order. That order also keeps the method signature line unchanged. I'd just move this above the `private` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707660458


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -181,11 +181,34 @@ public static Map<Integer, String> indexQuotedNameById(
     return indexer.byId();
   }
 
+  /**
+   * Creates a mapping from lower-case field names to their corresponding field IDs.
+   *
+   * <p>This method iterates over the fields of the provided struct and maps each field's name
+   * (converted to lower-case) to its ID. If two fields have the same lower-case name, an
+   * `IllegalArgumentException` is thrown.
+   *
+   * @param struct the struct type whose fields are to be indexed
+   * @return a map where the keys are lower-case field names and the values are field IDs
+   * @throws IllegalArgumentException if two fields have the same lower-case name

Review Comment:
   Minor: Should this be `IllegalArgumentException`? I think `IllegalStateException` may be more accurate because the state of the partition spec can't produce a case insensitive index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702020422


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -442,9 +456,10 @@ private Types.NestedField findSourceColumn(String sourceName) {
     Builder identity(String sourceName, String targetName) {
       Types.NestedField sourceColumn = findSourceColumn(sourceName);
       checkAndAddPartitionName(targetName, sourceColumn.fieldId());
+      String normalizedTargetName = caseSensitive ? targetName : targetName.toLowerCase();

Review Comment:
   Yes, I didn't intend to change the visibility of any of the methods. The new methods should all be private, though.
   
   I think that this should be done for all of the configuration methods that create a partition name (`targetName`) instead of normalizing to lower case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702992594


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -182,11 +181,10 @@ public static Map<Integer, String> indexQuotedNameById(
   }
 
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
-    Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
+    ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
     indexByName(struct)
-        .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
-    return indexByLowerCaseName;
+        .forEach((name, integer) -> builder.put(name.toLowerCase(Locale.ROOT), integer));
+    return builder.buildOrThrow();

Review Comment:
   I'll make the change but it kind of assumes that only two fields will ever have conflicting lowercase names. What if the schema contains fields named `data`, `DATA` and `Data`?
   
   (I agree having a schema with both `data` and `DATA` would be rare. Having a schema with `data`, `DATA` and `Data` would be another order of magnitude rare.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702994679


##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -182,11 +181,10 @@ public static Map<Integer, String> indexQuotedNameById(
   }
 
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
-    Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
+    ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
     indexByName(struct)
-        .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
-    return indexByLowerCaseName;
+        .forEach((name, integer) -> builder.put(name.toLowerCase(Locale.ROOT), integer));
+    return builder.buildOrThrow();

Review Comment:
   I have manually tested this change and the result is below
   ```
   java.lang.IllegalArgumentException: Unable to build field name to id mapping because two fields have the same lower case name: order_date and ORDER_DATE
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2229264518

   Case-Insensitivity is an engine behavior IMO, Iceberg provides some convenience functions to make dealing with such engines more simply but the underlying representation should always be case sensitive. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707712257


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,993 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "partition1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "partition1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketSourceNameAllowsExactDuplicateWhenCaseSensitive() {
+    Schema schema = SCHEMA_CASE_SENSITIVE;
+    PartitionSpec spec =
+        PartitionSpec.builderFor(schema)
+            .bucket("data", 10, "partition1")
+            .bucket("data", 10, "PARTITION1")
+            .build();
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "partition1", Types.IntegerType.get()),
+            NestedField.optional(1001, "PARTITION1", Types.IntegerType.get()));
+
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketTargetNameDefaultValueUsingCaseSensitiveSchema() {

Review Comment:
   The schema is not case sensitive or insensitive. I think the case that this is testing is when the partition name is derived from the source field.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) {
     }
 
     private void checkAndAddPartitionName(String name, Integer sourceColumnId) {
-      Types.NestedField schemaField = schema.findField(name);
+      Types.NestedField schemaField =

Review Comment:
   I already investigated this and the answer is I don't believe so. `findSourceColumn` will throw an exception if the column isn't found whereas `checkAndAddPartitionName` can still succeed even if the column isn't found. (See what was previously line 400 and is now line 402)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) {
     }
 
     private void checkAndAddPartitionName(String name, Integer sourceColumnId) {
-      Types.NestedField schemaField = schema.findField(name);
+      Types.NestedField schemaField =

Review Comment:
   I already investigated this and the answer is I don't believe so. `findSourceColumn` will throw an exception if the column isn't found whereas `checkAndAddPartitionName` can still succeed even if the column isn't found. (See line 480/482)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) {
     }
 
     private void checkAndAddPartitionName(String name, Integer sourceColumnId) {
-      Types.NestedField schemaField = schema.findField(name);
+      Types.NestedField schemaField =

Review Comment:
   I already investigated this and the answer is I don't believe so. `findSourceColumn` will throw an exception if the column isn't found whereas `checkAndAddPartitionName` can still succeed even if the column isn't found. (See line 400/402)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2248955661

   @RussellSpitzer @amogh-jahagirdar I understand your position on case sensitivity. Given that position it seems to me that PartitionSpec.java does not enable case-insensitivity because the current implementation of that class does not provide a way to call Schema.java's case-insensitive helper methods. For example the PartitionSpec.Builder.Identity method currently can only search for schema columns in a case-sensitive manner even though Schema offers case-insensitive methods for finding columns.
   
   One solution would be to add lots of method overloads, e.g. `PartitionSpec.Builder.Identity(String, String)` and `PartitionSpec.Builder.caseInsensitiveIdentity(String, String)` or `PartitionSpec.Builder.Identity(String, String, Boolean)`. This solution seems rather unattractive because it would increase the class's footprint and create noise thus making the class at least a little bit harder to grok.
   
   Or is the solution as simple as saying PartitionSpec and PartitionSpec.Builder are case-sensitive and there are no convenience methods for supporting case-insensitivity?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257156417

   I think that the API proposed here is the right approach.
   
   @amogh-jahagirdar makes a good point about `compatibleWith` and possibly other methods that we use to reason about partition specs. I think that it is important that we don't introduce case insensitivity as a property of the `PartitionSpec` itself, which would require a spec change, but still support the convenience that this PR introduces.
   
   To be more clear, I like this builder method because it can be used to make it easier for engines to configure tables. If the engine is case insensitive, the integration can pass that flag and the user's input to the `PARTITIONED BY` clause doesn't need to be pre-processed. But I think that we do need to ensure that we have consistent behavior for code paths that use the partition name that's based on the original column name. To do that, we just need to make sure that the partition name is derived from the column name in the schema, not the column name passed to the builder methods.
   
   That's not what the code does currently. For example, see [`identity(String)`](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L454). I think this PR will need to refactor the builder methods to use a consistent name if this introduces a code path where `sourceName` and `sourceField.name()` may not match.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696085669


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) {
     }
 
     private void checkAndAddPartitionName(String name, Integer sourceColumnId) {
-      Types.NestedField schemaField = schema.findField(name);
+      Types.NestedField schemaField =

Review Comment:
   should this call the `findSourceColumn` method?



##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   `caseInsensitiveFindField` uses normalized lower-case string for name -> id indexing.  who can ensure the schema don't have two fields with names like `data` and `DATA`? Otherwise, `caseInsensitiveFindField` search is ambiguous. I am wondering if the `caseSensitive` config need to be pushed into the `Schema` and spec?



##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;

Review Comment:
   current `caseSensitive(boolean)` seems more consistent with existing code, like `Scan` interface.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696005290


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -95,6 +95,19 @@ public void testSpecInfoPartitionedTable() {
         .doesNotContainKey(Integer.MAX_VALUE);
   }
 
+  @TestTemplate
+  public void testSpecInfoPartitionedTableCaseInsensitive() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(schema).caseSensitive(false).identity("DATA").build();

Review Comment:
   can you also add a test case where `caseSensitive(true)` and spec cannot be built?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702860252


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java:
##########
@@ -0,0 +1,697 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StructType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestPartitionSpecBuilderCaseSensitivity {
+
+  private static final int V2_FORMAT_VERSION = 2;
+  private static final Schema SCHEMA_CASE_INSENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "category", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "order_time", Types.TimestampType.withoutZone()),
+          required(6, "ship_date", Types.DateType.get()),
+          required(7, "ship_time", Types.TimestampType.withoutZone()));
+
+  private static final Schema SCHEMA_CASE_SENSITIVE =
+      new Schema(
+          required(1, "id", Types.IntegerType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "DATA", Types.StringType.get()),
+          required(4, "order_date", Types.DateType.get()),
+          required(5, "ORDER_DATE", Types.DateType.get()),
+          required(6, "order_time", Types.TimestampType.withoutZone()),
+          required(7, "ORDER_TIME", Types.TimestampType.withoutZone()));
+
+  @TempDir private Path temp;
+  private File tableDir = null;
+
+  @BeforeEach
+  public void setupTableDir() throws IOException {
+    this.tableDir = Files.createTempDirectory(temp, "junit").toFile();
+  }
+
+  @AfterEach
+  public void cleanupTables() {
+    TestTables.clearTables();
+  }
+
+  @Test
+  public void testPartitionTypeWithColumnNamesThatDifferOnlyInLetterCase() {
+    Schema schema =
+        new Schema(
+            required(1, "id", Types.IntegerType.get()),
+            required(2, "data", Types.StringType.get()),
+            required(3, "DATA", Types.StringType.get()),
+            required(4, "order_date", Types.DateType.get()));
+    PartitionSpec spec = PartitionSpec.builderFor(schema).identity("data").identity("DATA").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", schema, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(
+            NestedField.optional(1000, "data", Types.StringType.get()),
+            NestedField.optional(1001, "DATA", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testPartitionTypeWithIdentityTargetName() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(SCHEMA_CASE_INSENSITIVE).identity("data", "p1").build();
+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA_CASE_INSENSITIVE, spec, V2_FORMAT_VERSION);
+
+    StructType expectedType =
+        StructType.of(NestedField.optional(1000, "p1", Types.StringType.get()));
+    StructType actualType = Partitioning.partitionType(table);
+    assertThat(actualType).isEqualTo(expectedType);
+  }
+
+  @Test
+  public void testBucketSourceNameDoesNotAllowExactDuplicateWhenCaseSensitive() {
+    assertThatIllegalArgumentException()
+        .isThrownBy(
+            () ->
+                PartitionSpec.builderFor(SCHEMA_CASE_SENSITIVE)
+                    .bucket("data", 10, "p1")

Review Comment:
   nit: it is a bit difficult to read `p1` vs `P1`. use some other more distinguiable names, like `partition1` and `PARTITION1`. this applies to other tests too



##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -182,11 +181,10 @@ public static Map<Integer, String> indexQuotedNameById(
   }
 
   public static Map<String, Integer> indexByLowerCaseName(Types.StructType struct) {
-    Map<String, Integer> indexByLowerCaseName = Maps.newHashMap();
+    ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
     indexByName(struct)
-        .forEach(
-            (name, integer) -> indexByLowerCaseName.put(name.toLowerCase(Locale.ROOT), integer));
-    return indexByLowerCaseName;
+        .forEach((name, integer) -> builder.put(name.toLowerCase(Locale.ROOT), integer));
+    return builder.buildOrThrow();

Review Comment:
   Instead of depending on ImmutableMap builder for the dup check, I am wondering if we should handle the collision check here in order to throw an exception with more informative error msg than `IllegalArgumentException: Multiple entries with same key: a=2 and a=1`. E.g., the error msg can be sth like `Unable to build field name to id mapping because two fields have the same lower case name...`. We can also get the original field name since we know the two field ids and have the original schema/struct.
   
   also let's add Javadoc to this method to document the behavior under collision.



##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -95,6 +96,30 @@ public void testSpecInfoPartitionedTable() {
         .doesNotContainKey(Integer.MAX_VALUE);
   }
 
+  @TestTemplate
+  public void testSpecInfoPartitionedTableCaseInsensitive() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(schema).caseSensitive(false).identity("DATA").build();
+    TestTables.TestTable table = TestTables.create(tableDir, "test", schema, spec, formatVersion);
+
+    assertThat(table.spec()).isEqualTo(spec);
+    assertThat(table.spec().lastAssignedFieldId()).isEqualTo(spec.lastAssignedFieldId());
+    assertThat(table.specs())
+        .containsExactly(entry(spec.specId(), spec))
+        .doesNotContainKey(Integer.MAX_VALUE);

Review Comment:
   for my education, what is this assertion for?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1702976107


##########
core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java:
##########
@@ -95,6 +96,30 @@ public void testSpecInfoPartitionedTable() {
         .doesNotContainKey(Integer.MAX_VALUE);
   }
 
+  @TestTemplate
+  public void testSpecInfoPartitionedTableCaseInsensitive() {
+    PartitionSpec spec =
+        PartitionSpec.builderFor(schema).caseSensitive(false).identity("DATA").build();
+    TestTables.TestTable table = TestTables.create(tableDir, "test", schema, spec, formatVersion);
+
+    assertThat(table.spec()).isEqualTo(spec);
+    assertThat(table.spec().lastAssignedFieldId()).isEqualTo(spec.lastAssignedFieldId());
+    assertThat(table.specs())
+        .containsExactly(entry(spec.specId(), spec))
+        .doesNotContainKey(Integer.MAX_VALUE);

Review Comment:
   Are you asking about the specific assert below?  Actually, it doesn't matter. All I did was take the test above, `testSpecInfoPartitionedTable` on line 88, and make a case-insensitive version of the test. To understand the purpose of the assert you'll have to ask the person who wrote `testSpecInfoPartitionedTable`.
   
   ```
   assertThat(table.specs())
           .containsExactly(entry(spec.specId(), spec))
           .doesNotContainKey(Integer.MAX_VALUE);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1700713501


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   Oh, I see what I missed. It isn't using `ImmutableMap.Builder`. It is creating the lower case index with `Maps.newHashMap()`. I just read it wrong yesterday.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2264186832

   Initially this PR added support for case-insensitive for column names when creating a PartitionSpec. I have now added support for case-insensitive `targetName`s in each of the transform methods.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "sl255051 (via GitHub)" <gi...@apache.org>.
sl255051 commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1699234156


##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {
       dedupFields.put(dedupKey, field);
     }
 
+    public Builder caseSensitive(boolean sensitive) {
+      this.caseSensitive = sensitive;
+      return this;
+    }
+
     public Builder withSpecId(int newSpecId) {
       this.specId = newSpecId;
       return this;
     }
 
     private Types.NestedField findSourceColumn(String sourceName) {
-      Types.NestedField sourceColumn = schema.findField(sourceName);
+      Types.NestedField sourceColumn =
+          this.caseSensitive
+              ? schema.findField(sourceName)
+              : schema.caseInsensitiveFindField(sourceName);

Review Comment:
   @rdblue The timing of your answer is impeccable because at the very moment I received your answer I was debugging that very method, `TypeUtil.indexByLowerCaseName()`, to see how it handles `data` and `DATA`. The answer is that the method does not throw an exception at all.
   
   If you start with this schema:
   ```
   Schema schema = new Schema(
           required(1, "id", Types.IntegerType.get()),
           required(2, "data", Types.StringType.get()),
           required(3, "DATA", Types.StringType.get()));
   ```
   then calling `TypeUtil.indexByLowerCaseName(schema)` will return a RegularImmutableMap of
   ```
   {
       "data": 3,
       "id": 1
   }
   ```
   So either the result is undefined or (by implementation detail) is the ID of the last column with case-insensitive matching name. But the bottom line is there is no runtime failure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2299429899

   Looks good and I don't see any other open comments so I'll merge. Thanks, @sl255051!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707687316


##########
api/src/test/java/org/apache/iceberg/TestSchemaCaseSensitivity.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;
+
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestSchemaCaseSensitivity {
+
+  @Test
+  public void
+      testSchemaWithColumnNamesThatDifferOnlyInLetterCaseThrowsOnCaseInsensitiveFindField() {

Review Comment:
   Minor: I think these test case names can be more direct. How about `testCaseInsensitiveFieldCollision`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10678:
URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2229975315

   +1 to what @RussellSpitzer said. The spec by itself does not mandate case insensitivity for field names in a schema. SQL engines can enforce the case insensitivity that they desire but the metadata representation should probably be agnostic to all that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org