You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "sayedkeika (via GitHub)" <gi...@apache.org> on 2025/02/16 12:21:47 UTC
[PR] LDE Project - Extended I/O Framework: Readers/Writers for Parquet [systemds]
sayedkeika opened a new pull request, #2229:
URL: https://github.com/apache/systemds/pull/2229
### Overview
This pull request adds support for the Parquet file format. The implementation includes both readers and writers that can handle Parquet files sequentially and concurrently.
### Details
- Single-threaded reader and writer for smaller datasets
- Multi-threaded reader and writer for larger datasets
- Component tests for both sequential and parallel operations that cover different schemas
- Detailed documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extended I/O Framework: Readers/Writers for Parquet [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 closed pull request #2229: Extended I/O Framework: Readers/Writers for Parquet
URL: https://github.com/apache/systemds/pull/2229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extended I/O Framework: Readers/Writers for Parquet [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 commented on PR #2229:
URL: https://github.com/apache/systemds/pull/2229#issuecomment-2815535658
LGTM - thanks for the patch @sayedkeika. During the merge I resolved the merge conflicts, fixed the warnings and formatting (tabs over spaces), added additional tests (sparse data), and left a FIXME (for removing the ExampleParquetWriter). Additionally, I fixed the parallel write task to call the sequential write instead of the parallel writer again (which worked because the number of part files was 1 due to the small size).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org