You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "honno (via GitHub)" <gi...@apache.org> on 2023/03/29 17:03:38 UTC

[GitHub] [arrow] honno opened a new issue, #34774: Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs

honno opened a new issue, #34774:
URL: https://github.com/apache/arrow/issues/34774

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   It seems the interchange `Column.null_count()` ([relevant spec](https://github.com/data-apis/dataframe-api/blob/d10a096ccc1612ec4acf8503aff58b3acb4e3738/protocol/dataframe_protocol.py#L311-L313)) has erroneous behaviour
   
   ```python
   >>> import pyarrow as pa
   >>> pa.__version__
   '12.0.0.dev304'  # from https://pypi.fury.io/arrow-nightlies/
   >>> df = pa.table([pa.array([float("nan")], type=pa.float64())], ["foo"])
   >>> dfi = df.__dataframe__()
   >>> col = dfi.get_column(0)
   >>> col.null_count
   0  # should be 1
   ```
   
   I assume this is because Arrow does not treat NaNs as nulls, which semantically makes sense, but in the interchange protocol it should—see https://github.com/vaexio/vaex/issues/2120 for a related discussion.
   
   See pandas for expected behaviour
   
   ```python
   >>> import pandas as pa
   >>> df = pd.DataFrame({"foo": [float("nan")]})
   >>> dfi = df.__dataframe__()
   >>> col = dfi.get_column(0)
   >>> col.null_count
   1
   ```
   
   cc @AlenkaF (let me know if not to tag you on things! coincidentally I was working on https://github.com/data-apis/dataframe-interchange-tests/issues/20 today when Ralf commented heh.)
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] AlenkaF closed issue #34774: [Python] Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #34774: [Python] Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs
URL: https://github.com/apache/arrow/issues/34774


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] AlenkaF commented on issue #34774: [Python] Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #34774:
URL: https://github.com/apache/arrow/issues/34774#issuecomment-1493391945

   Closing this issue as we have received confirmation from the protocol side that `nan` and `null` can be treated separately, see discussion in https://github.com/data-apis/dataframe-api/issues/126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] AlenkaF commented on issue #34774: [Python] Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #34774:
URL: https://github.com/apache/arrow/issues/34774#issuecomment-1490151014

   > cc @AlenkaF (let me know if not to tag you on things! coincidentally I was working on [data-apis/dataframe-interchange-tests#20](https://github.com/data-apis/dataframe-interchange-tests/issues/20) today when Ralf commented heh.)
   
   Do tag me please! =)
   Very cool you both jumped on it so quickly hehe.
   
   > I assume this is because Arrow does not treat NaNs as nulls, which semantically makes sense, but in the interchange protocol it should—see [vaexio/vaex#2120](https://github.com/vaexio/vaex/issues/2120) for a related discussion.
   
   But we agree with Maarten 😊  https://github.com/vaexio/vaex/issues/2120#issuecomment-1194047890
   There would be loss of information for all the libraries that actually differentiate between nan and null, right? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org