You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kris-gaudel (via GitHub)" <gi...@apache.org> on 2025/07/24 03:49:42 UTC

[PR] #2123 Sanitize invalid Avro field names [iceberg-python]

kris-gaudel opened a new pull request, #2245:
URL: https://github.com/apache/iceberg-python/pull/2245

   <!--
   Thanks for opening a pull request!
   -->
   
   <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
   <!-- Closes #${GITHUB_ISSUE_ID} -->
   
   # Rationale for this change
   Fixing sanitization behaviour to match specification and Java implementation
   
   # Are these changes tested?
   Yes - Unit and integration tests
   
   # Are there any user-facing changes?
   Yes - Field names will be sanitized to be Avro compatible if not already
   
   <!-- In the case of user-facing changes, please add the changelog label. -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2260594293


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +524,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)
+
         result = {
-            "name": field.name,
-            "field-id": field.field_id,
+            "name": field_name,
+            FIELD_ID_PROP: field.field_id,
             "type": field_result if field.required else ["null", field_result],
         }
 
+        if not is_valid_field_name:
+            result[ICEBERG_FIELD_NAME_PROP] = orig_field_name

Review Comment:
   it was added to match the java implementation which is considered to be the reference implementation 
   
   im actually not sure where `ICEBERG_FIELD_NAME_PROP` is used, even in the java library. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2261041619


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +524,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)
+
         result = {
-            "name": field.name,
-            "field-id": field.field_id,
+            "name": field_name,
+            FIELD_ID_PROP: field.field_id,
             "type": field_result if field.required else ["null", field_result],
         }
 
+        if not is_valid_field_name:
+            result[ICEBERG_FIELD_NAME_PROP] = orig_field_name

Review Comment:
   I checked exactly the same thing when reviewing this, and came to the same conclusion 🤣  I think it is mostly for debugging purposes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3146167660

   @kevinjqliu Thanks for the feedback! In regards to the adding the spark integration test, I've been running into some issues with running integration tests locally that use Spark
   
   ```un.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available```
   
   Do you know what could be causing this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3157396322

   Ok so I downgraded my Java version and the integration tests run now, but I'm still having some issues with integration tests failing due to some spark issues. 
   
   Does CI run the integration test suite or just unit test suite? I have a feeling my changes are missing something...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2249128495


##########
tests/integration/test_writes/test_writes.py:
##########
@@ -1201,6 +1201,49 @@ def test_sanitize_character_partitioned(catalog: Catalog) -> None:
     assert len(tbl.scan().to_arrow()) == 22
 
 
+@pytest.mark.integration
+@pytest.mark.parametrize("catalog", [pytest.lazy_fixture("session_catalog")])
+def test_sanitize_character_partitioned_avro_bug(catalog: Catalog) -> None:
+    table_name = "default.test_table_partitioned_sanitized_character_avro"
+    try:
+        catalog.drop_table(table_name)
+    except NoSuchTableError:
+        pass
+
+    schema = Schema(
+        NestedField(id=1, name="😎", field_type=StringType(), required=False),
+    )
+
+    partition_spec = PartitionSpec(
+        PartitionField(
+            source_id=1,
+            field_id=1001,
+            transform=IdentityTransform(),
+            name="😎",
+        )
+    )
+
+    tbl = _create_table(
+        session_catalog=catalog,
+        identifier=table_name,
+        schema=schema,
+        partition_spec=partition_spec,
+        data=[
+            pa.Table.from_arrays(
+                [pa.array([str(i) for i in range(22)])], schema=pa.schema([pa.field("😎", pa.string(), nullable=False)])
+            )
+        ],
+    )
+
+    assert len(tbl.scan().to_arrow()) == 22
+
+    con = tbl.scan().to_duckdb("table_test_debug")
+    result = con.query("SELECT * FROM table_test_debug").fetchall()
+    assert len(result) == 22
+
+    assert con.query("SHOW table_test_debug").fetchone() == ("😎", "VARCHAR", "YES", None, None, None)

Review Comment:
   ah i tested the wrong thing in the reference PR. This is still using pyiceberg table scan to do the reading! We want to use duckdb to read instead.
   
   I had the fix in my local branch. This fails in main branch but is fixed with this PR
   
   ```suggestion
       # verify that we can read the table with DuckDB
       import duckdb
   
       location = tbl.metadata_location
       duckdb.sql("INSTALL iceberg; LOAD iceberg;")
       # Configure S3 settings for DuckDB to match the catalog configuration
       duckdb.sql("SET s3_endpoint='localhost:9000';")
       duckdb.sql("SET s3_access_key_id='admin';")
       duckdb.sql("SET s3_secret_access_key='password';")
       duckdb.sql("SET s3_use_ssl=false;")
       duckdb.sql("SET s3_url_style='path';")
       result = duckdb.sql(f"SELECT * FROM iceberg_scan('{location}')").fetchall()
       assert len(result) == 10
   ```



##########
tests/test_avro_sanitization.py:
##########


Review Comment:
   nit: looks like some of these tests are redundant, can we consolidate some of them? 
   
   these 3 are quiet similar, we can just add all the different pairs to the same test
   ```
   test_avro_field_name_sanitization
   test_edge_cases
   test_edge_cases_sanitization
   ```
   
   so are these 3
   ```
   test_avro_compatibility
   test_complex_schema_sanitization
   test_avro_schema_conversion_sanitization
   ```
   
   and we might be able to combine these 2
   ```
   test_avro_file_structure_verification
   test_emoji_field_name_sanitization
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3146217235

   > Do you know what could be causing this?
   
   Looks like CI passed, so it could be with your local setup. I put that in chatgpt, it says
   ```
   🧠 Root Causes:
   Running on a restricted JVM (like GraalVM Native Image or a non-Oracle/OpenJDK distribution)
   
   Using a Java version where these internals are restricted (like Java 17+ with strong encapsulation)
   
   Running in a context that limits reflection or unsafe access (e.g., with --illegal-access=deny)
   
   Spark compiled with a newer Java version but running on an incompatible JVM
   
   ```
   
   what java version are you using? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3156743229

   > > Do you know what could be causing this?
   > 
   > Looks like CI passed, so it could be with your local setup. I put that in chatgpt, it says
   > 
   > ```
   > 🧠 Root Causes:
   > Running on a restricted JVM (like GraalVM Native Image or a non-Oracle/OpenJDK distribution)
   > 
   > Using a Java version where these internals are restricted (like Java 17+ with strong encapsulation)
   > 
   > Running in a context that limits reflection or unsafe access (e.g., with --illegal-access=deny)
   > 
   > Spark compiled with a newer Java version but running on an incompatible JVM
   > ```
   > 
   > what java version are you using?
   
   I have Java 21 active, but I will dry downgrading my version and see if that fixes it. Thank you for looking into it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3156752194

   i didnt even know there's a java 22 already, haha
   thats likely the reason, iceberg java only supports up to 21
   https://github.com/apache/iceberg/blob/7ffc718d2857c1f4e4e7e1d70eebc8662020d6bd/build.gradle#L81


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3160885889

   nice! @kris-gaudel I think we're almost good to go
   
   Could you address these 2 comments when you get a chance? 
   https://github.com/apache/iceberg-python/pull/2245#discussion_r2249128495
   https://github.com/apache/iceberg-python/pull/2245#discussion_r2249130024


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3161003441

   @kevinjqliu Done! Lmk if I missed anything


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2246261560


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +532,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)
+
         result = {
-            "name": field.name,
-            "field-id": field.field_id,
+            "name": field_name,
+            FIELD_ID_PROP: field.field_id,
             "type": field_result if field.required else ["null", field_result],
         }
 
+        if not is_valid_field_name:
+            result[ICEBERG_FIELD_NAME_PROP] = orig_field_name

Review Comment:
   thanks for adding this, heres the relevant code from java
   https://github.com/apache/iceberg/blob/1bd8d5e2de56d05180030b856ce2c50c66ef1f13/core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java#L117-L120



##########
tests/integration/test_avro_compatibility.py:
##########
@@ -0,0 +1,335 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import tempfile
+
+import pytest
+from fastavro import reader
+
+import pyiceberg.avro.file as avro
+from pyiceberg.io.pyarrow import PyArrowFileIO
+from pyiceberg.schema import Schema
+from pyiceberg.typedef import Record
+from pyiceberg.types import IntegerType, NestedField, StringType
+from pyiceberg.utils.schema_conversion import AvroSchemaConversion
+
+
+class AvroTestRecord(Record):
+    """Test record class for Avro compatibility testing."""
+
+    @property
+    def valid_field(self) -> str:
+        return self._data[0]
+
+    @property
+    def invalid_field(self) -> int:
+        return self._data[1]
+
+    @property
+    def field_with_dot(self) -> str:
+        return self._data[2]
+
+    @property
+    def field_with_hash(self) -> int:
+        return self._data[3]
+
+    @property
+    def field_starting_with_digit(self) -> str:
+        return self._data[4]
+
+
+@pytest.mark.integration
+def test_avro_compatibility() -> None:
+    """Test that Avro files with sanitized names can be read by other tools."""
+
+    schema = Schema(
+        NestedField(field_id=1, name="valid_field", field_type=StringType(), required=True),
+        NestedField(field_id=2, name="invalid.field", field_type=IntegerType(), required=True),
+        NestedField(field_id=3, name="field_with_dot", field_type=StringType(), required=True),
+        NestedField(field_id=4, name="field_with_hash", field_type=IntegerType(), required=True),

Review Comment:
   nit: are these suppose to have `.` and `#`? 
   
   field_id=2 also took care of the `.` case
   



##########
pyiceberg/schema.py:
##########
@@ -1391,7 +1409,9 @@ def _sanitize_name(name: str) -> str:
 
 
 def _sanitize_char(character: str) -> str:
-    return "_" + character if character.isdigit() else "_x" + hex(ord(character))[2:].upper()
+    if character.isdigit():
+        return "_" + character
+    return "_x" + hex(ord(character))[2:].upper()

Review Comment:
   reviewer note, this is the same implementation, just refactored



##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +532,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)

Review Comment:
   nit: `make_compatible_name` already calls `is_valid_field_name` 
   https://github.com/apache/iceberg-python/blob/904c0b77c33768716e8672b38bae73bfcf565fbf/pyiceberg/schema.py#L1357-L1361
   
   we can simply this part
   ```suggestion
           field_name = make_compatible_name(orig_field_name)
   ```
   



##########
tests/integration/test_avro_compatibility.py:
##########


Review Comment:
   thanks for adding these tests! one nit is that these aren't "integration tests" per se. Integration tests in pyiceberg rely on the docker infra (for spark/iceberg rest catalog/hive/etc).
   
   Integration tests are more difficult to run because they require setup. 
   
   I think none of these tests rely on any infra, can we move them to regular tests? maybe in `test_avro_sanitization.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3141238591

   following up to https://github.com/apache/iceberg-python/issues/2123#issuecomment-3140972785 and https://github.com/apache/iceberg-python/issues/2123#issuecomment-2995265399
   
   I see that spark produces `_xD83D_xDE0E`. pyiceberg previously produces `\uD83D\uDE0E` but now produces `_x1F60E`.
   it might be a good idea to add an integration test for pyiceberg reading from spark written table and spark reading from pyiceberg written table


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "nvartolomei (via GitHub)" <gi...@apache.org>.
nvartolomei commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2260340907


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +524,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)
+
         result = {
-            "name": field.name,
-            "field-id": field.field_id,
+            "name": field_name,
+            FIELD_ID_PROP: field.field_id,
             "type": field_result if field.required else ["null", field_result],
         }
 
+        if not is_valid_field_name:
+            result[ICEBERG_FIELD_NAME_PROP] = orig_field_name

Review Comment:
   Do we know when this is used? Did we add it only because we know Java implementation does it or we have an actual reasoning for why this is necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2261291351


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +524,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)
+
         result = {
-            "name": field.name,
-            "field-id": field.field_id,
+            "name": field_name,
+            FIELD_ID_PROP: field.field_id,
             "type": field_result if field.required else ["null", field_result],
         }
 
+        if not is_valid_field_name:
+            result[ICEBERG_FIELD_NAME_PROP] = orig_field_name

Review Comment:
   Lol I saw the constant used in the Java library so I mirrored it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2249040283


##########
tests/integration/test_avro_compatibility.py:
##########
@@ -0,0 +1,335 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import tempfile
+
+import pytest
+from fastavro import reader
+
+import pyiceberg.avro.file as avro
+from pyiceberg.io.pyarrow import PyArrowFileIO
+from pyiceberg.schema import Schema
+from pyiceberg.typedef import Record
+from pyiceberg.types import IntegerType, NestedField, StringType
+from pyiceberg.utils.schema_conversion import AvroSchemaConversion
+
+
+class AvroTestRecord(Record):
+    """Test record class for Avro compatibility testing."""
+
+    @property
+    def valid_field(self) -> str:
+        return self._data[0]
+
+    @property
+    def invalid_field(self) -> int:
+        return self._data[1]
+
+    @property
+    def field_with_dot(self) -> str:
+        return self._data[2]
+
+    @property
+    def field_with_hash(self) -> int:
+        return self._data[3]
+
+    @property
+    def field_starting_with_digit(self) -> str:
+        return self._data[4]
+
+
+@pytest.mark.integration
+def test_avro_compatibility() -> None:
+    """Test that Avro files with sanitized names can be read by other tools."""
+
+    schema = Schema(
+        NestedField(field_id=1, name="valid_field", field_type=StringType(), required=True),
+        NestedField(field_id=2, name="invalid.field", field_type=IntegerType(), required=True),
+        NestedField(field_id=3, name="field_with_dot", field_type=StringType(), required=True),
+        NestedField(field_id=4, name="field_with_hash", field_type=IntegerType(), required=True),

Review Comment:
   I'll remove these redundant fields



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: sanitize invalid Avro field names in manifest file [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#discussion_r2246271818


##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -524,12 +532,19 @@ def field(self, field: NestedField, field_result: AvroType) -> AvroType:
         if isinstance(field_result, dict) and field_result.get("type") == "record":
             field_result["name"] = f"r{field.field_id}"
 
+        orig_field_name = field.name
+        is_valid_field_name = _valid_avro_name(orig_field_name)
+        field_name = orig_field_name if is_valid_field_name else make_compatible_name(orig_field_name)

Review Comment:
   nit: `make_compatible_name` already calls `_valid_avro_name ` 
   https://github.com/apache/iceberg-python/blob/904c0b77c33768716e8672b38bae73bfcf565fbf/pyiceberg/schema.py#L1357-L1361
   
   we can simply this part
   ```suggestion
           field_name = make_compatible_name(orig_field_name)
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] #2123 Sanitize invalid Avro field names [iceberg-python]

Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2245:
URL: https://github.com/apache/iceberg-python/pull/2245#issuecomment-3141121767

   @kevinjqliu When you have a moment could you review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org