You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "theelderbeever (via GitHub)" <gi...@apache.org> on 2023/09/28 19:58:51 UTC

[GitHub] [arrow-datafusion] theelderbeever opened a new issue, #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

theelderbeever opened a new issue, #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686

   ### Describe the bug
   
   Performing a SQL query against a NDJson with partition columns will fail when filtering on any of the partition columns with the following error. In this case my partition column is a timestamp but it holds for other types as well.
   
   > ArrowError(JsonError("Encountered unmasked nulls in non-nullable StructArray child: Field { name: \"hourly_timestamp\", data_type: Timestamp(Second, None), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }"))
   
   It correctly prunes the files however, it doesn't populate the partition predicate correctly. This is in contrast to the ParquetExec which adds an extra predicate to populate the partition column.
   
   > JsonExec: file_groups={1 group: [[Users/taylorbeever/git/theelderbeever/df-test/data/ndjson/hourly_timestamp=2023-09-25T20:00:00/data.ndjson]]}, projection=[id, timestamp, value, hourly_timestamp]
   
   > ParquetExec: file_groups={1 group: [[Users/taylorbeever/git/theelderbeever/df-test/data/parquet/hourly_timestamp=2023-09-25T20:00:00/data.zstd.parquet]]}, projection=[id, timestamp, value, hourly_timestamp], predicate=hourly_timestamp@3 = 1695672000
   
   Attempted solutions - all fail:
   - Add partition columns to each json file.
   - Define the Schema for the table
   - Other datatypes for partition
   
   ### To Reproduce
   
   I created an example repo [here](https://github.com/theelderbeever/datafusion-ndjson-issue).
   
   Example data is included in the repo. All code is contained in `src/main.rs`. The parquet files are identical data to ndjson files.  They do not contain a column for the partition column as written.
   
   To run:
   
   First the parquet one which will succeed.
   ```console
   RUST_LOG=debug cargo run -- parquet
   ```
   
   Then the ndjson which will fail.
   ```console
   RUST_LOG=debug cargo run -- ndjson
   ```
   
   
   ### Expected behavior
   
   Partitioned table reads shouldn't fail when filtering on a partition column.
   
   Additionally, the default file_extension for `NDJsonReadOptions` is `.json` which is a little misleading. Its should be one of `.ndjson` or `.jsonl`.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741282810

   @alamb I wouldn't mind jumping in on this but it might be a little while until I can. Have some deadlines to hit first 👎 ....
   
   From what I could find looking into the `NDJsonExec` and `ParquetExec` implementations it looked like `NDJsonExec` just plain doesn't have any logic for handling the predicates whereas parquet does. Do it could end up being a pretty invasive update. I am pretty new to datafusion though so I don't fully understand how the plans are created.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741538080

   I wouldn't go as far as saying that it can't handle them because we are actually using them with partitions to solid success right now. 
   
   Could logic be added to "hydrate" the partition column after the file read? I need to dig further in to understand what that even means however, since you know about the qualifying files during the scan you could maybe parse the file path to generate the column on the fly and add it post reading parsing the files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741272003

   Thank you for the report @theelderbeever 
   
   It would be great if somone had time to help debug this and figure out what is wrong


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1741327569

   >  just plain doesn't have any logic for handling the predicates whereas parquet d
   
   Yes this is correct -- I don't think the NDJsonExec can really do much to push predicates down the way Parquet can. So maybe we just have to fix the way it handles partitioned tables (or change the code so  it doesn't claim to handle them)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] NDJsonExec doesn't properly apply predicates on partitioned tables. [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.
URL: https://github.com/apache/arrow-datafusion/issues/7686


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] theelderbeever commented on issue #7686: NDJsonExec doesn't properly apply predicates on partitioned tables.

Posted by "theelderbeever (via GitHub)" <gi...@apache.org>.
theelderbeever commented on issue #7686:
URL: https://github.com/apache/arrow-datafusion/issues/7686#issuecomment-1740056657

   Worth noting that since #7670 was merged this only applies to the `SELECT *` case from what I can tell. With the projection removed when not in the select statement there is a workaround for many cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org