You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/02/03 02:25:06 UTC
[GitHub] [arrow] manojkarthick opened a new pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
manojkarthick opened a new pull request #9401:
URL: https://github.com/apache/arrow/pull/9401
Add method `compressed_size()` to get compressed column data size in a row group. The JVM [parquet-mr](https://github.com/apache/parquet-mr) library provides a method to get the same [here](https://github.com/apache/parquet-mr/blob/0a4e3eea991f7588c9c5e056e9d7b32a76eed5da/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/metadata/BlockMetaData.java#L114) . This would help in calculating the total uncompressed and compressed size of the Parquet file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] manojkarthick commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
manojkarthick commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772644573
> I wonder if it would be possible to add a test for this API (mostly so that someone doesn't delete it by accident thinking it isn't used)?
Thank you @alamb ! Could you provide any pointers on what kind of test would be relevant here? I thought of adding a test, but couldn't come up with one that is useful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] manojkarthick commented on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
manojkarthick commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773291600
Thanks a lot @alamb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772840091
(you can run clippy locally via `cargo clippy --all-targets --workspace -- -D warnings -A clippy::redundant_field_names`) if you want to run it the same as we run it in CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773229279
Thanks again @manojkarthick 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772184423
https://issues.apache.org/jira/browse/ARROW-11479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb edited a comment on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773229279
Thanks again @manojkarthick 👍 -- I just merged this PR to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772759543
@manojkarthick How about something along the lines of https://github.com/apache/arrow/blob/8a134013494eb7b15004ea7bba8bca8a9562ffdd/rust/parquet/src/file/metadata.rs#L656
The test could create a row group metadata with two columns and assert that `compressed_size()` returns the sum of the columns' size?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb edited a comment on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773229279
Thanks again @manojkarthick 👍 -- I just merged this PR to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9401:
URL: https://github.com/apache/arrow/pull/9401
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773229279
Thanks again @manojkarthick 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] manojkarthick commented on pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
manojkarthick commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-773291600
Thanks a lot @alamb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772839380
Thanks @manojkarthick !
It looks like there is a small linter failure now [link](https://github.com/apache/arrow/pull/9401/checks?check_run_id=1825802194):
```
error: redundant clone
--> parquet/src/file/metadata.rs:757:68
|
757 | let row_group_meta = RowGroupMetaData::builder(schema_descr.clone())
| ^^^^^^^^ help: remove this
|
= note: `-D clippy::redundant-clone` implied by `-D warnings`
note: this value is dropped without further use
--> parquet/src/file/metadata.rs:757:56
|
757 | let row_group_meta = RowGroupMetaData::builder(schema_descr.clone())
| ^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone
error: aborting due to previous error
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] manojkarthick commented on pull request #9401: ARROW-11479: Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
manojkarthick commented on pull request #9401:
URL: https://github.com/apache/arrow/pull/9401#issuecomment-772849489
> (you can run clippy locally via `cargo clippy --all-targets --workspace -- -D warnings -A clippy::redundant_field_names`) if you want to run it the same as we run it in CI
Thanks @alamb ! I've fixed the clippy failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9401: ARROW-11479: [Rust] [Parquet] Add Method to get compressed size of columns from row group metadata
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9401:
URL: https://github.com/apache/arrow/pull/9401
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org