You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/10/26 18:04:48 UTC
[I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
andygrove opened a new issue, #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040
### What is the problem the feature request solves?
Comet has native code for decoding Parquet structures into Arrow arrays. This issue is for discussing delegating to the [parquet crate](https://crates.io/crates/parquet) instead for these operations.
The benefits of this approach include:
- Support for complex types. The parquet crate already supports reading maps and structs. We could implement the same support in the Comet native code but it is probably a lot of work
- Support for StringView and benefitting from related performance optimizations (see [1] and [2] for details)
- Benefit from ongoing optimization work and active community
- Reduce maintenance efforts in Comet
Possible downsides of this approach:
- Lose the performance benefit of re-using mutable buffers? (although this also comes with a maintenance cost)
[1] https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/
[2] https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2452032756
> Hmm, I guess that doc is either out-of-dated or inaccurate. The Parquet supports complex types.
Yes, there is `StructArrayReader`, for example:
```rust
pub struct StructArrayReader {
children: Vec<Box<dyn ArrayReader>>,
data_type: ArrowType,
struct_def_level: i16,
struct_rep_level: i16,
nullable: bool,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2447977412
> * Support for complex types. The parquet crate already supports reading maps and structs.
@andygrove The doc here https://crates.io/crates/parquet says the reader supports ` Primitive column value readers`. Doesn't say whether complex types are supported.
Also, something to keep in mind, Comet has Spark specific decoding (for instance Int96 timestamps https://github.com/apache/datafusion-comet/blob/00cf79b253d9fa31dcc0506facbaf7b2dac25cdd/native/core/src/parquet/read/values.rs#L841). Some post processing may need to be done unless we can hook in our own decoder for special cases like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1040: Use parquet crate for decoding Parquet data into Arrow arrays
URL: https://github.com/apache/datafusion-comet/issues/1040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2448016119
Hmm, I guess that doc is either out-of-dated or inaccurate. The Parquet supports complex types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2627796948
Closing this since we now have code in main for using ParquetExec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2448002002
As a first step, we can certainly switch to the lower level `file` APIs to replace the Comet FileReader. `https://docs.rs/parquet/latest/parquet/file/serialized_reader/struct.SerializedPageReader.html` would do very nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2719140742
FWIW we have begun implementing encryption support upstream in arrow-rs / parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use parquet crate for decoding Parquet data into Arrow arrays [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1040:
URL: https://github.com/apache/datafusion-comet/issues/1040#issuecomment-2452043015
I created a Google doc where we can collaborate on design:
https://docs.google.com/document/d/1eiDFEScPjxBMahJW6lmBI8JjVlI6CwhiJgkTSsTvPVY/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org