You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/02/13 15:28:55 UTC
[GitHub] [arrow] Fokko opened a new issue, #34162: [Python] `is_null(nan_is_null=True)` does not work with only NaN's
Fokko opened a new issue, #34162:
URL: https://github.com/apache/arrow/issues/34162
### Describe the bug, including details regarding any error messages, version, and platform.
I was working on some test-cases for the PyIceberg integration, and hit this edge case. When you have a file with only NaN values, it will be skipped when reading the file with a `is_null(nan_is_null=True)` filter.
In Spark, I create the following table:
```sql
CREATE TABLE test_null_nan
USING iceberg
AS SELECT
1 AS idx,
float('NaN') AS col_numeric
UNION ALL SELECT
2 AS idx,
null AS col_numeric
UNION ALL SELECT
3 AS idx,
1 AS col_numeric
```
This then creates three files with each one record:
```
➜ python git:(fd-integration-tests) ✗ pyiceberg --catalog local files default.test_null_nan
Snapshots: local.default.test_null_nan
└── Snapshot 870844541941792785, schema 0: s3a://warehouse/wh/default/test_null_nan/metadata/snap-870844541941792785-1-a05e1621-f735-4837-bb86-ce9886da3e6b.avro
└── Manifest: s3a://warehouse/wh/default/test_null_nan/metadata/a05e1621-f735-4837-bb86-ce9886da3e6b-m0.avro
├── Datafile: s3a://warehouse/wh/default/test_null_nan/data/00000-0-658408d0-d063-4caa-b310-f68552713bea-00001.parquet
├── Datafile: s3a://warehouse/wh/default/test_null_nan/data/00001-1-5e625fcb-4a0c-4082-9371-7f4897768ccd-00001.parquet
└── Datafile: s3a://warehouse/wh/default/test_null_nan/data/00002-2-11de56ee-27c1-45ff-be61-cc52727c1b84-00001.parquet
```
If I filter using `pc.col('col_numeric').is_null(nan_is_null=True) & ~pc.col('col_numeric').is_null()` I don't get any results. When I rewrite the table into a single file:
```sql
CREATE TABLE test_null_nan_rewritten
USING iceberg
AS SELECT * FROM test_null_nan
```
And then do the same filter operation, I do get results. I suspect there is something off with the page skipping when `nan_is_null=True`.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34162: [Python] `is_null(nan_is_null=True)` does not work with only NaN's
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34162:
URL: https://github.com/apache/arrow/issues/34162#issuecomment-1444552350
> Maybe because it skips the page when doing predicate pushdown?
That seems likely. I did a quick test and `SimplifyWithGuarantee` is not considering `nan_is_null` so `is_null(x, true)` given the guarantee `is_null(x, false)` simplifies to `literal(true)` and it should not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34162: [Python] `is_null(nan_is_null=True)` does not work with only NaN's
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34162:
URL: https://github.com/apache/arrow/issues/34162#issuecomment-1433256003
@Fokko I am trying to reproduce this with just pyarrow, but for now not succeeding.
First, just checking if plain filtering on in memory data works (which it does):
```
>>> table = pa.table({"idx": [1, 2, 3], "col_numeric": [np.nan, None, 1]})
>>> table
pyarrow.Table
idx: int64
col_numeric: double
----
idx: [[1,2,3]]
col_numeric: [[nan,null,1]]
>>> table.filter(pc.field('col_numeric').is_null(nan_is_null=True) & ~pc.field('col_numeric').is_null())
pyarrow.Table
idx: int64
col_numeric: double
----
idx: [[1]]
col_numeric: [[nan]]
```
Then, if I write this file to Parquet and ensure to get each row in one row group (still in a single file), reading it with a filter should use the row group statistics for pruning as a first step. This also seems to work:
```
>>> pq.write_table(table, "test_filter_nan.parquet", row_group_size=1)
>>> meta = pq.read_metadata("test_filter_nan.parquet")
>>> meta.num_row_groups
3
>>> pq.read_table("test_filter_nan.parquet", filters=pc.field('col_numeric').is_null(nan_is_null=True) & ~pc.field('col_numeric').is_null())
pyarrow.Table
idx: int64
col_numeric: double
----
idx: [[1]]
col_numeric: [[nan]]
```
Now, maybe this depends on how the Parquet file was written. When written with pyarrow as above, the row groups with the values NaN and null don't have statistics set (and so won't never be skipped or not because of predicate pushdown rowgroup filtering).
@Fokko Your files were created with Spark, I assume? Would it be possible to share those 3 small parquet files from your example above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Fokko commented on issue #34162: [Python] `is_null(nan_is_null=True)` does not work with only NaN's
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #34162:
URL: https://github.com/apache/arrow/issues/34162#issuecomment-1441663151
@jorisvandenbossche Thanks for diving into this. Sorry for the late reply, I had to shift priorities a bit, but now I'm back.
Looking at the notebook:
When running `.filter()` it works well, but when passing the filter to the `read_table` messes things up. Maybe because it skips the page when doing predicate pushdown?
The parquet files: [parquet-files.zip](https://github.com/apache/arrow/files/10813386/parquet-files.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org