(iceberg-python) branch main updated: feat: support pyarrow float16 by widening to float on read/write (#3590)

[fo...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

Fokko pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git


The following commit(s) were added to refs/heads/main by this push:
     new 4223f4cb1 feat: support pyarrow float16 by widening to float on read/write (#3590)
4223f4cb1 is described below

commit 4223f4cb187e796befd61c2536e31ada4a9aa631
Author: Anas Khan <an...@gmail.com>
AuthorDate: Sun Jul 19 11:31:13 2026 +0530

    feat: support pyarrow float16 by widening to float on read/write (#3590)
    
    # Rationale for this change
    
    PyArrow `float16` (`halffloat`) currently raises
    `UnsupportedPyArrowTypeException` during schema conversion, because
    `_ConvertToIceberg.primitive` only handles `float32` and `float64`:
    
    ```python
    >>> import pyarrow as pa
    >>> from pyiceberg.io.pyarrow import _ConvertToIceberg, visit_pyarrow
    >>> visit_pyarrow(pa.float16(), _ConvertToIceberg())
    pyiceberg.exceptions.UnsupportedPyArrowTypeException: Column 'x' has an unsupported type: halffloat
    ```
    
    Iceberg has no half-precision float, but `float16 -> float32` is
    **lossless**: every IEEE 754 half value (including the maximum finite
    value 65504) is exactly representable in single precision. This mirrors
    how the same method already widens `int8`/`int16` to `IntegerType`, and
    how `ArrowProjectionVisitor._cast_if_needed` already widens smaller
    integers up for cross-platform compatibility. Mapping `float16 ->
    FloatType` is the float analogue, so `float16` columns round-trip
    instead of erroring.
    
    Changes (`pyiceberg/io/pyarrow.py`):
    
    - `_ConvertToIceberg.primitive`: map `pa.float16()` -> `FloatType()`.
    - `ArrowProjectionVisitor._cast_if_needed`: widen smaller float types to
    the target type on write (parallel to the existing integer-widening
    branch), so `float16` arrays are cast to `float32`. Narrowing falls
    through to the existing `promote()` handling.
    
    No dependency changes; `pyproject.toml` / `uv.lock` are untouched and
    the imports used were already present.
    
    A note on a design choice, deferring to maintainers: widening `float16`
    silently (rather than erroring or gating behind a config flag) follows
    the existing `int8`/`int16 -> Integer` precedent. Happy to gate it
    behind a config option instead if you'd prefer. The new float-widening
    branch also makes `float32 -> DoubleType` actually cast the array
    (parallel to int widening), so it slightly tightens float promotion in
    general, not just `float16`.
    
    ## Are these changes tested?
    
    Yes:
    
    - `tests/io/test_pyarrow_visitor.py::test_pyarrow_float16_to_iceberg`
    asserts the schema mapping `pa.float16() -> FloatType()`.
    - `tests/io/test_pyarrow.py::test__to_requested_schema_float_promotion`
    is parametrized over `f16 -> Float`, `f16 -> Double`, and `f32 ->
    Double`, asserting both the written PyArrow type and that the values are
    preserved.
    
    Both pass locally, the surrounding visitor suite and the sibling
    integer-promotion test still pass, and `make lint` (ruff, ruff-format,
    mypy, pydocstyle, codespell, uv-lock) is clean. The integration suite
    (Docker/Spark) was not run locally.
    
    ## Are there any user-facing changes?
    
    Yes. PyArrow tables with `float16` columns can now be converted/written
    through PyIceberg (they map to Iceberg `float` and are stored as
    `float32`), where they previously raised
    `UnsupportedPyArrowTypeException`. This is purely additive; existing
    `float32`/`float64` behavior is unchanged.
---
 pyiceberg/io/pyarrow.py          | 12 ++++++++++++
 tests/io/test_pyarrow.py         | 29 +++++++++++++++++++++++++++++
 tests/io/test_pyarrow_visitor.py |  6 ++++++
 3 files changed, 47 insertions(+)

diff --git a/pyiceberg/io/pyarrow.py b/pyiceberg/io/pyarrow.py
index 7e1babd58..f05781d9b 100644
--- a/pyiceberg/io/pyarrow.py
+++ b/pyiceberg/io/pyarrow.py
@@ -1438,6 +1438,9 @@ class _ConvertToIceberg(PyArrowSchemaVisitor[IcebergType | Schema]):
             else:
                 # Does not exist (yet)
                 raise TypeError(f"Unsupported integer type: {primitive}")
+        elif pa.types.is_float16(primitive):
+            # Iceberg has no half-precision float; widen to single precision (lossless)
+            return FloatType()
         elif pa.types.is_float32(primitive):
             return FloatType()
         elif pa.types.is_float64(primitive):
@@ -1979,6 +1982,15 @@ class ArrowProjectionVisitor(SchemaWithPartnerVisitor[pa.Array, pa.Array | None]
                         target_width = target_type.bit_width
                         if source_width < target_width:
                             return values.cast(target_type)
+                elif isinstance(field.field_type, (FloatType, DoubleType)):
+                    # Cast smaller float types to target type for cross-platform compatibility
+                    # Only allow widening conversions (smaller bit width to larger), e.g. float16 -> float32
+                    # Narrowing conversions fall through to promote() handling below
+                    if pa.types.is_floating(values.type):
+                        source_width = values.type.bit_width
+                        target_width = target_type.bit_width
+                        if source_width < target_width:
+                            return values.cast(target_type)
 
             if field.field_type != file_field.field_type:
                 target_schema = schema_to_pyarrow(
diff --git a/tests/io/test_pyarrow.py b/tests/io/test_pyarrow.py
index 532311899..dde354735 100644
--- a/tests/io/test_pyarrow.py
+++ b/tests/io/test_pyarrow.py
@@ -3109,6 +3109,35 @@ def test__to_requested_schema_integer_promotion(
     assert result.column(0).to_pylist() == [1, 2, 3, None]
 
 
+@pytest.mark.parametrize(
+    "arrow_type,iceberg_type,expected_arrow_type",
+    [
+        (pa.float16(), FloatType(), pa.float32()),
+        (pa.float16(), DoubleType(), pa.float64()),
+        (pa.float32(), DoubleType(), pa.float64()),
+    ],
+)
+def test__to_requested_schema_float_promotion(
+    arrow_type: pa.DataType,
+    iceberg_type: PrimitiveType,
+    expected_arrow_type: pa.DataType,
+) -> None:
+    """Test that smaller float types are cast to target Iceberg type during write."""
+    requested_schema = Schema(NestedField(1, "col", iceberg_type, required=False))
+    file_schema = requested_schema
+
+    arrow_schema = pa.schema([pa.field("col", arrow_type)])
+    data = pa.array([1.5, 2.25, 3.0, None], type=arrow_type)
+    batch = pa.RecordBatch.from_arrays([data], schema=arrow_schema)
+
+    result = _to_requested_schema(
+        requested_schema, file_schema, batch, downcast_ns_timestamp_to_us=False, include_field_ids=False
+    )
+
+    assert result.schema[0].type == expected_arrow_type
+    assert result.column(0).to_pylist() == [1.5, 2.25, 3.0, None]
+
+
 def test_pyarrow_file_io_fs_by_scheme_cache() -> None:
     # It's better to set up multi-region minio servers for an integration test once `endpoint_url` argument
     # becomes available for `resolve_s3_region`
diff --git a/tests/io/test_pyarrow_visitor.py b/tests/io/test_pyarrow_visitor.py
index 6727b8c76..e98d76e26 100644
--- a/tests/io/test_pyarrow_visitor.py
+++ b/tests/io/test_pyarrow_visitor.py
@@ -128,6 +128,12 @@ def test_pyarrow_int64_to_iceberg() -> None:
     assert visit(converted_iceberg_type, _ConvertToArrowSchema()) == pyarrow_type
 
 
+def test_pyarrow_float16_to_iceberg() -> None:
+    pyarrow_type = pa.float16()
+    converted_iceberg_type = visit_pyarrow(pyarrow_type, _ConvertToIceberg())
+    assert converted_iceberg_type == FloatType()
+
+
 def test_pyarrow_float32_to_iceberg() -> None:
     pyarrow_type = pa.float32()
     converted_iceberg_type = visit_pyarrow(pyarrow_type, _ConvertToIceberg())