You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "theelderbeever (via GitHub)" <gi...@apache.org> on 2023/09/28 19:58:51 UTC
[GitHub] [arrow-datafusion] theelderbeever opened a new issue, #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
theelderbeever opened a new issue, #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686
### Describe the bug
Performing a SQL query against a NDJson with partition columns will fail when filtering on any of the partition columns with the following error. In this case my partition column is a timestamp but it holds for other types as well.
> ArrowError(JsonError("Encountered unmasked nulls in non-nullable StructArray child: Field { name: \"hourly_timestamp\", data_type: Timestamp(Second, None), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }"))
It correctly prunes the files however, it doesn't populate the partition predicate correctly. This is in contrast to the ParquetExec which adds an extra predicate to populate the partition column.
> JsonExec: file_groups={1 group: [[Users/taylorbeever/git/theelderbeever/df-test/data/ndjson/hourly_timestamp=2023-09-25T20:00:00/data.ndjson]]}, projection=[id, timestamp, value, hourly_timestamp]
> ParquetExec: file_groups={1 group: [[Users/taylorbeever/git/theelderbeever/df-test/data/parquet/hourly_timestamp=2023-09-25T20:00:00/data.zstd.parquet]]}, projection=[id, timestamp, value, hourly_timestamp], predicate=hourly_timestamp@3 = 1695672000
Attempted solutions - all fail:
- Add partition columns to each json file.
- Define the Schema for the table
- Other datatypes for partition
### To Reproduce
I created an example repo [here](https://github.com/theelderbeever/datafusion-ndjson-issue).
Example data is included in the repo. All code is contained in `src/main.rs`. The parquet files are identical data to ndjson files. They do not contain a column for the partition column as written.
To run:
First the parquet one which will succeed.
```console
RUST_LOG=debug cargo run -- parquet
```
Then the ndjson which will fail.
```console
RUST_LOG=debug cargo run -- ndjson
```
### Expected behavior
Partitioned table reads shouldn't fail when filtering on a partition column.
Additionally, the default file_extension for `NDJsonReadOptions` is `.json` which is a little misleading. Its should be one of `.ndjson` or `.jsonl`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741282810
@alamb I wouldn't mind jumping in on this but it might be a little while until I can. Have some deadlines to hit first 👎 ....
From what I could find looking into the `NDJsonExec` and `ParquetExec` implementations it looked like `NDJsonExec` just plain doesn't have any logic for handling the predicates whereas parquet does. Do it could end up being a pretty invasive update. I am pretty new to datafusion though so I don't fully understand how the plans are created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741538080
I wouldn't go as far as saying that it can't handle them because we are actually using them with partitions to solid success right now.
Could logic be added to "hydrate" the partition column after the file read? I need to dig further in to understand what that even means however, since you know about the qualifying files during the scan you could maybe parse the file path to generate the column on the fly and add it post reading parsing the files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741272003
Thank you for the report @theelderbeever
It would be great if somone had time to help debug this and figure out what is wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741327569
> just plain doesn't have any logic for handling the predicates whereas parquet d
Yes this is correct -- I don't think the NDJsonExec can really do much to push predicates down the way Parquet can. So maybe we just have to fix the way it handles partitioned tables (or change the code so it doesn't claim to handle them)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NDJsonExec doesn't properly apply predicates on partitioned tables. [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
URL: https://github.com/apache/arrow-datafusion/issues/7686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1740056657
Worth noting that since #7670 was merged this only applies to the `SELECT *` case from what I can tell. With the projection removed when not in the select statement there is a workaround for many cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org