You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ldsantos0911 (via GitHub)" <gi...@apache.org> on 2025/06/18 21:53:20 UTC
[I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
ldsantos0911 opened a new issue, #2119:
URL: https://github.com/apache/iceberg-python/issues/2119
### Question
```
import pyarrow as pa
schema = pa.schema(
[
pa.field("col1", pa.string(), nullable=True),
]
)
# table is a Table with the above schema
df = pa.Table.from_pylist(
[
{"col1": None}
]
)
table.overwrite(df)
```
In the above example, we encounter an error like this `UnsupportedPyArrowTypeException: Column 'col1' has an unsupported type: null`, with underlying cause
```
in _ConvertToIceberg.primitive(self, primitive)
1211 return FixedType(primitive.byte_width)
-> 1213 raise TypeError(f"Unsupported type: {primitive}")
TypeError: Unsupported type: null
```
Is there any reason we wouldn't want to support the case where pyarrow has marked a Field as `null`? As a workaround/fix, I was thinking that we could exclude `pa.null()` Fields in `visit_pyarrow(obj: pa.StructType, visitor: PyArrowSchemaVisitor[T])`. This way, the column would effectively be missing and any required/nullable enforcement would be performed accordingly. Would this have any undesired consequences?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #2119: When writing data from a PyArrow DataFrame, how should we handle 'null' Fields?
URL: https://github.com/apache/iceberg-python/issues/2119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3196878435
@Fokko Sorry for the wait, still working on it but got caught up with work. PR up this week!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3091281827
heres a workaround https://github.com/apache/iceberg-python/pull/2155#pullrequestreview-3035006048
we should document pyarrow null type. I've encountered this before :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3196749680
@kris-gaudel Gentle ping, are you still working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3168798148
@kris-gaudel done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3207574635
@ldsantos0911 already handled the tests and the core functionality. I made the error more descriptive so developers know the expected behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? [iceberg-python]
Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on issue #2119:
URL: https://github.com/apache/iceberg-python/issues/2119#issuecomment-3166123762
Can I be assigned this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org