You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/09 13:33:16 UTC
[GitHub] [arrow-rs] tustvold opened a new pull request, #1827: Refactor parquet::arrow module
tustvold opened a new pull request, #1827:
URL: https://github.com/apache/arrow-rs/pull/1827
# Which issue does this PR close?
Closes #.
# Rationale for this change
As the reader logic in particular has gotten more sophisticated, the module has gotten rather large. This moves some files around to make the structure easier to understand
# What changes are included in this PR?
* Creates a new buffer module where the arrow compat logic can live
* Renames files to `module_name/mod.rs` instead of having `module_name.rs` and `module_name/`
*
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#discussion_r893618642
##########
parquet/benches/arrow_reader.rs:
##########
@@ -355,27 +355,6 @@ fn create_string_byte_array_dictionary_reader(
.unwrap()
}
-fn create_complex_object_byte_array_dictionary_reader(
Review Comment:
ComplexObjectArrayReader is no longer used and is slated for removal (#1661 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#discussion_r893513381
##########
parquet/src/arrow/mod.rs:
##########
@@ -122,14 +122,12 @@
experimental_mod!(array_reader);
pub mod arrow_reader;
pub mod arrow_writer;
-mod bit_util;
+mod buffer;
#[cfg(feature = "async")]
pub mod async_reader;
-experimental_mod!(converter);
Review Comment:
This module is slated for eventual removal as part of #1661
It is marked experimental so hiding it from the public API is not a breaking change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#issuecomment-1151212473
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1827](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7b76fd) into [master](https://codecov.io/gh/apache/arrow-rs/commit/8f1fd128e387fb2ccf8d74ef3e542e4455f1c6b9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8f1fd12) will **decrease** coverage by `0.00%`.
> The diff coverage is `50.00%`.
```diff
@@ Coverage Diff @@
## master #1827 +/- ##
==========================================
- Coverage 83.45% 83.45% -0.01%
==========================================
Files 200 200
Lines 56697 56696 -1
==========================================
- Hits 47315 47313 -2
- Misses 9382 9383 +1
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [parquet/src/arrow/array\_reader/builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyYXlfcmVhZGVyL2J1aWxkZXIucnM=) | `93.50% <ø> (ø)` | |
| [parquet/src/arrow/array\_reader/byte\_array.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyYXlfcmVhZGVyL2J5dGVfYXJyYXkucnM=) | `85.71% <ø> (ø)` | |
| [parquet/src/arrow/arrow\_reader.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyb3dfcmVhZGVyLnJz) | `96.87% <ø> (ø)` | |
| [parquet/src/arrow/arrow\_writer/levels.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyb3dfd3JpdGVyL2xldmVscy5ycw==) | `96.92% <ø> (ø)` | |
| [parquet/src/arrow/arrow\_writer/mod.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyb3dfd3JpdGVyL21vZC5ycw==) | `97.53% <ø> (ø)` | |
| [parquet/src/arrow/buffer/bit\_util.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYnVmZmVyL2JpdF91dGlsLnJz) | `100.00% <ø> (ø)` | |
| [parquet/src/arrow/buffer/converter.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYnVmZmVyL2NvbnZlcnRlci5ycw==) | `83.33% <ø> (ø)` | |
| [parquet/src/arrow/buffer/dictionary\_buffer.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYnVmZmVyL2RpY3Rpb25hcnlfYnVmZmVyLnJz) | `94.44% <ø> (ø)` | |
| [parquet/src/arrow/buffer/offset\_buffer.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYnVmZmVyL29mZnNldF9idWZmZXIucnM=) | `94.79% <ø> (ø)` | |
| [parquet/src/arrow/mod.rs](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvbW9kLnJz) | `44.44% <ø> (ø)` | |
| ... and [6 more](https://codecov.io/gh/apache/arrow-rs/pull/1827/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8f1fd12...a7b76fd](https://codecov.io/gh/apache/arrow-rs/pull/1827?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] Ted-Jiang commented on pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#issuecomment-1152079088
> Yes, the high level ticket is #1163. I'm hoping to pick it up soon, but large amounts of this API are public so needs some careful thought
Great! 👍 After implement send row_selection in `arrow_reader::get_record_reader_by_columns`
I would glad to help! 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#discussion_r893514396
##########
parquet/src/arrow/buffer/mod.rs:
##########
@@ -0,0 +1,23 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Logic for reading data into arrow buffers
+
+pub mod bit_util;
+pub mod offset_buffer;
+pub mod dictionary_buffer;
+pub mod converter;
Review Comment:
I would like to eventually upstream many of these into arrow-rs proper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #1827: Refactor parquet::arrow module
Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #1827:
URL: https://github.com/apache/arrow-rs/pull/1827#issuecomment-1152033356
> Is there any plan for simplify the reader API
Yes, the high level ticket is #1163. I'm hoping to pick it up soon, but large amounts of this API are public so needs some careful thought
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org