You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "JoaoAparicio (via GitHub)" <gi...@apache.org> on 2023/04/03 00:14:52 UTC
[GitHub] [arrow-julia] JoaoAparicio opened a new pull request, #412: Add kwarg to filter columns
JoaoAparicio opened a new pull request, #412:
URL: https://github.com/apache/arrow-julia/pull/412
Currently we don't have the option to load just a subset of the columns. This matters e.g. when compression is the bottleneck.
For example, create a compressed arrow file.
```julia
using Arrow
p = tempname();
N = 1000000
tbl = (
a=rand(N),
b=rand(N),
c=rand(N),
d=rand(N),
e=rand(N),
f=[rand(rand(0:100)) for _ in 1:N],
);
Arrow.write(p, tbl; compress=:zstd);
```
Column `f` is the longest - it has an expected 50*N elements vs N for the rest Some times we only care for some of the other columns. Currently we must decompress all columns regardless:
```julia
using BenchmarkTools
@btime tbl = Arrow.Table(p); # 359.205 ms (530 allocations: 794.23 MiB)
```
With this commit we can load only some of the columns
```julia
@btime tbl = Arrow.Table(p; filtercolumns=["a"]); # 6.146 ms (231 allocations: 14.33 MiB)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-3086039724
We need to rebase on main to proceed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1793267084
## [Codecov](https://app.codecov.io/gh/apache/arrow-julia/pull/412?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#412](https://app.codecov.io/gh/apache/arrow-julia/pull/412?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bc9169e) into [main](https://app.codecov.io/gh/apache/arrow-julia/commit/787768fc60ff09ea52be0c5fb7c40e04be47619b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (787768f) will **decrease** coverage by `82.45%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## main #412 +/- ##
==========================================
- Coverage 87.45% 5.00% -82.45%
==========================================
Files 26 25 -1
Lines 3283 3279 -4
==========================================
- Hits 2871 164 -2707
- Misses 412 3115 +2703
```
| [Files](https://app.codecov.io/gh/apache/arrow-julia/pull/412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [src/table.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL3RhYmxlLmps) | `0.00% <0.00%> (-92.50%)` | :arrow_down: |
... and [23 files with indirect coverage changes](https://app.codecov.io/gh/apache/arrow-julia/pull/412/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/codecov/) today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "Yuan-Ru-Lin (via GitHub)" <gi...@apache.org>.
Yuan-Ru-Lin commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-3081652435
Hi, what's the status of this PR? Would love to see what I can do @JoaoAparicio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] JoaoAparicio commented on pull request #412: Add kwarg to filter columns
Posted by "JoaoAparicio (via GitHub)" <gi...@apache.org>.
JoaoAparicio commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1493483023
https://github.com/apache/arrow-julia/issues/340
https://github.com/apache/arrow-julia/issues/353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "JoaoAparicio (via GitHub)" <gi...@apache.org>.
JoaoAparicio commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1826303596
Does anyone wanna re-run CI? Looks like macos got stuck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-3083870711
for the API, `filtercolumns` seems a bit ambiguous to me (are we keeping them or removing them). CSV.jl has `select` and `drop` (https://csv.juliadata.org/stable/reading.html#CSV.File) which seems nice. Maybe `select` or `select_columns` for the name here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1827023484
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] JoaoAparicio commented on pull request #412: Add kwarg to filter columns
Posted by "JoaoAparicio (via GitHub)" <gi...@apache.org>.
JoaoAparicio commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1496366774
Converting this to draft as I'm working on something that will supersede this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add kwarg to filter columns [arrow-julia]
Posted by "oschulz (via GitHub)" <gi...@apache.org>.
oschulz commented on PR #412:
URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-3083668945
This would be a very important feature for us, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org