You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/04 18:19:43 UTC
[GitHub] [arrow-datafusion] jorgecarleitao opened a new pull request #68: Experimenting with arrow2
jorgecarleitao opened a new pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68
I have been experimenting using arrow2 & parquet2 as backends for DataFusion. This is WIP and does not compile, but I would like to give some visibility to this work.
So far, I was able to keep all functionality, with a guaranteed increase in security and potentially some performance.
Goals:
* compile
* tests pass
* re-write readers and writers to leverage parallelism of both `parquet2` and the CSV reader in arrow2
* ???
* profit
-------------
Some notes:
* I removed `CastOptions` because casting does not need to be fallible; we can make any non-castable null and recover the set of failed casts from the differences in validity between the original array and the casted array, if so we wish.
* most kernels in Arrow2 return a `Box<dyn Array>` [for some reasons](https://github.com/jorgecarleitao/arrow2/tree/main/src/compute#design); we use `result.into()` to convert to `Arc`. This is very cheap and the best that arrow has to offer without the unstable channel.
* There are some changes in `min` and `max` for floats. Essentially, `arrow2` guarantees that the comparison operator used in `sort` for floats is the same as the one used in `min/max`, which required this small change due to `Ord` for floats still be part of unstable rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1012680203
closing this PR in favor of https://github.com/apache/arrow-datafusion/pull/1556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922238494
Here the full table, thanks to @jorgecarleitao for implementing the parquet change so quickly.
(output in ms, average of 3 runs )
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 2071.80 | 2267.46 |
| 3 | 1615.33 | 1817.35 |
| 5 | 2990.24 | 3222.48 |
| 6 | 883.02 | 1061.64 |
| 8 | OOM | OOM |
| 9 | OOM | OOM |
| 10 | 2784.24 | 3096.37 |
| 12 | 1473.96 | 2039.88 |
| 13 | 6207.32 | 5965.40 |
We can see arrow is almost always faster than the arrow2 branch.
Now, loaded into memory, it looks much more equal/mixed:
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 1775.07 | 1640.60 |
| 3 | 1128.12 | 1109.23 |
| 5 | 2528.62 | 2413.35 |
| 6 | 215.58 | 211.19 |
| 10 | 1886.62 | 2259.54 |
| 12 | 739.42 | 820.00 |
| 13 | 6397.59 | 6326.59 |
It seems some portion of the change could be that loading parquet is slower in arrow2, somehow.
E.g. loading the full lineitem table (16 files, 2.4GB total) takes ~8380 ms in arrow2 vs ~6484 ms in the master branch using the `-m` option in the benchmarking tool.
@jorgecarleitao any idea of why this might happen? I could do some profiling if that's useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-921923054
I got one query running on a patched version of DataFusion / the benchmark on SF=10 / Parquet data source (split evenly in 16 partitions):
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 6 | 883.02 | 1061.64 |
As you can see, this query runs a bit slower than the arrow-based version - didn't do any profiling or checking yet, but could be some primitives like hashing could be faster in the master branch right now.
If this is implemented
https://github.com/jorgecarleitao/arrow2/issues/418
I can probably get some more numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917611641
> Since we will finally move to arrow2/parquet2, at the same time, it might be quite a lot of work for the current parquet to adapt for async reading. I suppose I shouldn't waste @alamb time on this.
Moving datafusion to arrow2 definitely sounds like an approach to getting an async parquet reader. Depending on how trickey that proves, another approach might be to follow the path blazed by @jorgecarleitao in
https://github.com/jorgecarleitao/arrow2/pull/260 (that adds parquet reading to arrow2) to add async reading to the existing parquet crate. I tried to write up some thoughts on https://github.com/apache/arrow-rs/issues/111#issue-867656782
> I can devote non-trivial amounts of time to working on this PR, accelerating its maturity, and finally getting merged. @houqp could you share the progress currently and assign me anything that you think necessary? Thanks!
That is awesome @yjshen ! Thank you again for all your help and contribution in this area
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918553501
FYI I was trying to run the TPC-H benchmarks in datafusion against a current.
I had to add support for parquet (de)compression via datafusion:
```
-arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex"] }
+arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex", "io_parquet_compression"] }
```
And some small changes in the TPCH tool to make it compile.
The Parquet files generated by the benchmark were not readable yet though - this should be implemented here.
https://github.com/jorgecarleitao/arrow2/pull/402
Would be fun to have a full running benchmark against the same files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918020460
> @yjshen thanks for offering the help. Really appreciate it! I have pushed my WIP commit to https://github.com/houqp/arrow-datafusion/tree/arrow2-merge. This branch merged with a relatively new commit 7932cb9. I am still working to get it to build. My branch also requires a patched version of arrow2 at https://github.com/houqp/arrow2/tree/qp_ord (pending PR jorgecarleitao/arrow2#398). I currently have arrow2 patched to point to my local check out in Cargo.toml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-844480669
```
cargo build --lib
...
Finished dev [unoptimized + debuginfo] target(s)
```
π
I temporarily deactivated the predicate pushdown for parquet, but since parquet2 is ~3-20x faster, I do not see it as a deal breaker. The tests are not yet compiling; that is the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r717438333
##########
File path: .github/workflows/rust.yml
##########
@@ -310,8 +310,7 @@ jobs:
run: |
cargo miri setup
cargo clean
- # Ignore MIRI errors until we can get a clean run
- cargo miri test || true
+ cargo miri test
Review comment:
β€οΈ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-929750339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r707057736
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
Filed https://github.com/apache/arrow-datafusion/issues/996 to track the optimization work as a separate task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642554087
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
FYI this is a (extremely) bad way to combine hashes - I also learned this while improving the hash join!
E.g. see here: http://myeyesareblind.com/2017/02/06/Combine-hash-values/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-934192805
@jorgecarleitao Do you need any additional help/support to port Avro complex data structures ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-929750339
> arrow 6.0 is likely to be released a few weeks from now in mid october, so if we targeted arrow 7.0 (Jan / Feb 2022) as the timeline to converge that would let us get DataFusion over to arrow2 in a few weeks, and then have several months to work out the kinks prior to a real arrow 7.0 release.
Obviously Jorge is the best person to answer this question, so please take my words with a grain of salt :)
I am not actively working on arrow2 at the moment other than filing annoying issues ;) But from the activities I have observed in the arrow2 repo so far, it probably won't be ready for 1.x release after the arrow 6 release. Jorge is still releasing multiple versions of arrow2 a week, so I am concerned that moving to ASF at its current stage would slow him down unless we start to get significant more contributions to the arrow2 repo. Assuming we won't be able to get arrow2 into ASF in time, I propose we maintain an arrow2 branch in datafusion to buy us more time and reduce downstream impact. I can help maintain that branch and merge with master one or twice a month until arrow2 is ready. If we don't want to pay the cost of maintaining a separate arrow2 datafusion branch, we could also temporarily have datafusion depend the current arrow2 repo with the goal of eventually merging arrow2 with arrow as it matures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-978987996

(Command `cargo run --release --bin tpch -- benchmark datafusion --iterations 5 --path [path] --format parquet --query [query] --batch-size 8192 -p 16`, SF=10 dataset)
arrow2 | arrow
-- | --
1414,39 | 2994,92
1010,86 | 2826,7
1951,55 | 3871,32
504,08 | 1987,25
1778 | 3399,45
792,82 | 2470,72
2536,1 | 2611,67
Great work @jorgecarleitao @houqp @yjshen and others :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007923402
No worries, I was just eager to try out the potential improvements and
learn the new API !
Le sam. 8 janv. 2022 Γ 04:26, QP Hou ***@***.***> a Γ©crit :
> Apologize for the late reply, still catching up to all the goodness
> happened during the break. I should have communicated this better to save
> @Igosuki <https://github.com/Igosuki> some work :( But really impressive
> result, thank you for doing the benchmark!
>
> I have squashed our latest round of merge fixes into a single commit to
> keep things clean and pushed it to this PR. What is missing right now from
> my point of view is migrating the python crate and avro. @Igosuki
> <https://github.com/Igosuki> do you mind sending your avro changes to my
> branch at https://github.com/houqp/arrow-datafusion/tree/arrow2_merge so
> I can merge it into this PR? After that I think this PR is ready for a
> formal review with another round of quick master merge to pick up the
> removal of the python binding.
>
> @alamb <https://github.com/alamb> to answer your question, I think the
> arrow2 branch is ready to be tested more broadly now. I feel like we should
> merge this either to master or to an official arrow2 branch soon without
> waiting for arrow2 1.x release, which is probably going to take a while.
> The API in arrow2 is still evolving fast right now (in a good direction),
> so I don't see it getting stablized any time soon. For example, it just
> recently removed the RecoardBatch abstraction entirely, which also aligns
> with our goal to come up with our own recordbatch abstraction in
> datafusion: #1248 <https://github.com/apache/arrow-datafusion/issues/1248>
>
> β
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007877042>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADDFBREKA2MO6LSGD7KP43UU6VH3ANCNFSM43RKTGGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-913048130
That is great, @houqp π Let me know if you would like me to take any chunk of it, if you need any change in arrow2 for this, or any other help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-924611413
I also created the [arrow2 milestone](https://github.com/apache/arrow-datafusion/milestone/3) to help keep track of todo items, feel free to add more to the list as you see fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r711172838
##########
File path: datafusion/Cargo.toml
##########
@@ -49,8 +49,8 @@ force_hash_collisions = []
[dependencies]
ahash = "0.7"
hashbrown = { version = "0.11", features = ["raw"] }
-arrow = { version = "^5.2", features = ["prettyprint"] }
-parquet = { version = "^5.2", features = ["arrow"] }
+arrow = { package = "arrow2", version="0.5", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex"] }
Review comment:
Can we add `io_parquet_compression` here - parquet is not as useful as with compression :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 676.98 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1241.67 | 1219.40 |
| 6 | 92.38 | 97.44 |
| 10 | 1057.55 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 2114.55 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 15% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917611641
> Since we will finally move to arrow2/parquet2, at the same time, it might be quite a lot of work for the current parquet to adapt for async reading. I suppose I shouldn't waste @alamb time on this.
Moving datafusion to arrow2 definitely sounds like an approach to getting an async parquet reader. Depending on how trickey that proves, another approach might be to follow the path blazed by @jorgecarleitao in
https://github.com/jorgecarleitao/arrow2/pull/260 (that adds async parquet reading to arrow2) to add async reading to the existing parquet crate. I tried to write up some thoughts on https://github.com/apache/arrow-rs/issues/111#issuecomment-917611415
> I can devote non-trivial amounts of time to working on this PR, accelerating its maturity, and finally getting merged. @houqp could you share the progress currently and assign me anything that you think necessary? Thanks!
That is awesome @yjshen ! Thank you again for all your help and contribution in this area
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-916337521
This is pretty sweet -- thanks for taking it on @houqp !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r706567452
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
Got it, the optimization makes total sense to me. The protobuf ser/der is indeed a tricky problem. I am thinking it's probably better to track this optimization as a separate PR/task to reduce the scope of this PR. There are some other complications introduced from recent changes in datafusions as well including newly added `Hash` and `PartialOrd` traits implementation to the ScalarValue type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-930034589
> But from the activities I have observed in the arrow2 repo so far
This is great information -- thank you for sharing it @houqp .
> so I am concerned that moving to ASF at its current stage would slow him down unless we start to get significant more contributions to the arrow2 repo.
Yeah - it is definitely a 2 edged sword -- I suspect moving arrow2 to ASF would indeed slow @jorgecarleitao down but also would encourage more contributions. Balancing that tradeoff is trickey for sure.
> If we don't want to pay the cost of maintaining a separate arrow2 datafusion branch, we could also temporarily have datafusion depend the current arrow2 repo with the goal of eventually merging arrow2 with arrow as it matures.
Since datafusion relies on arrow, if we change it to rely on arrow2, any downstream projects that rely on datafusion that want to use the latest datafusion will also have to change to use arrow2 (to feed data in the `MemoryProvider`, for example). I think this is what you mean by "downstream impact"
Thus I would prefer we continue datafusion with a stable implementation of arrow -- and we can wait until that describes arrow2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922307204
Currently my intuition is that the parquet files generated by the TPC-H convert + `parquet` are somehow less optimal for `parquet2` and `arrow2` than they are for`parquet`.
I tried to check this by also generating parquet files via this branch (which uses `parquet2`) but getting some errors when reading statistics (even when enabling statistics in the writer). The files generated are somewhat smaller (~8%) - so there is at least some difference.
I added the parquet (lineitem) files over here:
https://drive.google.com/drive/folders/19kUhg_6o8PuEOwCCLGQq0zh1bAq2TzG0?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 725.57 | 675.02 |
| 3 | 507.22 | 489.79 |
| 5 | 1352.82 | 1219.40 |
| 6 | 109.01 | 97.44 |
| 10 | 1102.23 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 2114.55 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 10% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-979874642
Ah, it's *exactly* #1010 that introduces the regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642554876
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
Simple example is that combining two same hashes always map to `0` - regardless of which value they have and in which order they appear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-898847230
> Has anyone seen similar efforts for other libraries by a chance?
datafuse, a fork of datafusion, migrated to arrow2 in [this PR](https://github.com/datafuselabs/datafuse/pull/1239); polars keeps a [branch of arrow2](https://github.com/pola-rs/polars/compare/master...arrow2) (which it compiles the Python binaries from).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923086839
Discussion on the parquet results lead to following conclusion:
* quite a bit of CPU time is spent in utf8 validation (around 20% for a full lineitem parquet file or ~50% for a string column). We can use simdutf8 (https://github.com/jorgecarleitao/arrow2/pull/426) to speed up utf8 validation for >= 64 byte columns, especially for non-ascii data, but that doesn't help with smaller strings in the TPC-H files. It seems `parquet` might not or not always do the utf8 validation https://github.com/apache/arrow-rs/issues/786 . Beyond this `parquet2` is having a larger advantage with optional fields compared to `parquet`. Based on profiling, I think there are still some opportunities for improving the `parquet2` performance a bit more, but I couldn't find a lot of low hanging fruit.
Also there are some arrow2 updates which might have an effect on DataFusion queries and I will rerun the benchmarks when merged:
* https://github.com/jorgecarleitao/arrow2/pull/428 : makes ahash the default and adds some multiversioning to select SIMD instructions (aes / sse / etc. on x86_64) used by `aHash`. Note that `aHash` was not used by default before, this already gives a 2-3x boost over the Rust default `SipHash`. A nightly compiler might be even faster as it can use specialization in that case.
* https://github.com/jorgecarleitao/arrow2/pull/427 : improves performance on utf8 kernels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922259223
Thank you so much for this. π
I am also investigating since this does not match previous runs nor microbenchmarks. So, there is likely a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923086839
Discussion on the parquet results lead to following conclusions:
* quite a bit of CPU time is spent in utf8 validation (around 20% for a full lineitem parquet file or ~50% for a string column). We can use simdutf8 (https://github.com/jorgecarleitao/arrow2/pull/426) to speed up utf8 validation for >= 64 byte columns, especially for non-ascii data, but that doesn't help with smaller strings in the TPC-H files. It seems `parquet` might not or not always do the utf8 validation https://github.com/apache/arrow-rs/issues/786 . Beyond this `parquet2` is having a larger advantage with optional fields compared to `parquet`. Based on profiling, I think there are still some opportunities for improving the `parquet2` performance a bit more, but I couldn't find a lot of low hanging fruit.
Also there are some arrow2 updates which might have an effect on DataFusion queries and I will rerun the benchmarks when merged:
* https://github.com/jorgecarleitao/arrow2/pull/428 : makes ahash the default and adds some multiversioning to select SIMD instructions (aes / sse / etc. on x86_64) used by `aHash`. Note that `aHash` was not used by default before, this already gives a 2-3x boost over the Rust default `SipHash`. A nightly compiler might be even faster as it can use specialization in that case.
* https://github.com/jorgecarleitao/arrow2/pull/427 : improves performance on utf8 kernels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-931883876
> 1. The feature set is "close enough" to the current implementation of arrow that all DataFusion tests pass
> 2. There is a clear path, ability and willingness to help close any other gaps between the existing arrow-rs implementation
> 3. There is a community committed to its development and maintenance, under some sort of formal community project governance model (e.g. Apache Arrow)
I think this PR has proven arrow2 has passed `1`. For `2`, I think arrow2 meets the requirement as well. For `3`, we might need to give Jorge a bit more time to figure out the best governance model for the project and the community. Overall, I think things are looking great so far and I am seeing the end of the tunnel ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-930412418
> Thus I would prefer we continue datafusion with a stable implementation of arrow -- and we can wait until that describes arrow2.
Yes, this is what meant by downstream impact and I agree with you on this one. Making datafusion work with arrow2 is another big validation for the approach that arrow2 is taking. We just need to wait for the API to stablize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-913898374
@jorgecarleitao we need to build the row group filter lambda using parquet file metadata. I noticed the `RecordReader` is already reading that metadata in https://github.com/jorgecarleitao/arrow2/blob/92e227706c1235ddf3ef62dcba440df241711f48/src/io/parquet/read/record_batch.rs#L40
Is there a way for us to reuse what has already been read there to build the row group filter? `RecordReader::try_new` requires passing in a row group filter argument, so I read the metadata manually before calling `try_new`, I would be reading the same bytes twice.
This is for migrating https://github.com/apache/arrow-datafusion/blob/7932cb9373192ce2754b39c1f82f22c8a56b7266/datafusion/src/physical_plan/parquet.rs#L539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642571782
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
In the hash join this `combine_hashes` function is used, which also could be used here as well:
```rust
fn combine_hashes(l: u64, r: u64) -> u64 {
let hash = (17 * 37u64).wrapping_add(l);
hash.wrapping_mul(37).wrapping_add(r)
}
```
e.g. something like: `*hash = combine_hashes(hash, other)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-844480669
```
cargo build --lib
...
Finished dev [unoptimized + debuginfo] target(s)
```
π
I temporarily deactivated the predicate pushdown for parquet, but since parquet2 is ~3-20x faster, I do not see it as a deal breaker. The `like` was also deactivated for a moment. The tests are not yet compiling; that is the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-842643054
A couple of more updates pushed; still no compilation, but -1k LOC without loss of functionality, which is always nice.
fwiw, I plan to split this in smaller parts as there are useful regardless of whether this lands or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007923402
No worries, I was just eager to try out the potential improvements and
learn the new API !
Le sam. 8 janv. 2022 Γ 04:26, QP Hou ***@***.***> a Γ©crit :
> Apologize for the late reply, still catching up to all the goodness
> happened during the break. I should have communicated this better to save
> @Igosuki <https://github.com/Igosuki> some work :( But really impressive
> result, thank you for doing the benchmark!
>
> I have squashed our latest round of merge fixes into a single commit to
> keep things clean and pushed it to this PR. What is missing right now from
> my point of view is migrating the python crate and avro. @Igosuki
> <https://github.com/Igosuki> do you mind sending your avro changes to my
> branch at https://github.com/houqp/arrow-datafusion/tree/arrow2_merge so
> I can merge it into this PR? After that I think this PR is ready for a
> formal review with another round of quick master merge to pick up the
> removal of the python binding.
>
> @alamb <https://github.com/alamb> to answer your question, I think the
> arrow2 branch is ready to be tested more broadly now. I feel like we should
> merge this either to master or to an official arrow2 branch soon without
> waiting for arrow2 1.x release, which is probably going to take a while.
> The API in arrow2 is still evolving fast right now (in a good direction),
> so I don't see it getting stablized any time soon. For example, it just
> recently removed the RecoardBatch abstraction entirely, which also aligns
> with our goal to come up with our own recordbatch abstraction in
> datafusion: #1248 <https://github.com/apache/arrow-datafusion/issues/1248>
>
> β
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007877042>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADDFBREKA2MO6LSGD7KP43UU6VH3ANCNFSM43RKTGGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are ~~18~~ 13 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are 18 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007877042
Apologize for the late reply, still catching up to all the goodness happened during the break. I should have communicated this better to save @Igosuki some work :( But really impressive result, thank you for doing the benchmark!
I have squashed our latest round of merge fixes into a single commit to keep things clean and pushed it to this PR. What is missing right now from my point of view is migrating the python crate and avro. @Igosuki do you mind sending your avro changes to my branch at https://github.com/houqp/arrow-datafusion/tree/arrow2_merge so I can merge it into this PR? After that I think this PR is ready for a formal review with another round of quick master merge to pick up the removal of the python binding.
@alamb to answer your question, I think the arrow2 branch is ready to be tested more broadly now. I feel like we should merge this either to master or to an official arrow2 branch soon without waiting for arrow2 1.x release, which is probably going to take a while. The API in arrow2 is still evolving fast right now (in a good direction), so I don't see it getting stablized any time soon. For example, it just recently removed the RecoardBatch abstraction entirely, which also aligns with our goal to come up with our own recordbatch abstraction in datafusion: https://github.com/apache/arrow-datafusion/issues/1248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1009529577
@Igosuki, @yjshen already ported decimal support into https://github.com/houqp/arrow-datafusion/tree/arrow2_merge. Thanks for helping with this, I will wait for your PR before sending a new PR to create our official arrow2 branch :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004031259
This is very cool work @Igosuki π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-979870472
I confirmed that an older version (before https://github.com/apache/arrow-datafusion/pull/1010) doesn't have the problem (query 6 finishes in 813.29ms vs 2470ms).
Major Changes since that version:
* Parquet reading / listing code
* Older arrow code.
* ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922246917
Rerunning the benchmarks with `-p 16` now. Some profiling revealed option was added recently to have a maximum partitioning, which defaults to 2 in the benchmark.
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 1409.01 | 1355.47 |
| 3 | 1045.94 | 1047.49 |
| 5 | 1848.22 | 1847.41 |
| 6 | 403.26 | 450.15 |
| 10 | 1639.10 | 1886.21 |
| 12 | 2508.39 | 2596.48 |
| 13 | 2441.99 | 2296.75 |
Looks to be closer now than when only using 2 threads to load the data.
Skipping parquet (loading parquet is also slower when using 16 threads vs 2: 2708ms in arrow2 vs 2192 in master)
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 725.57 | 739.89 |
| 3 | 507.22 | 501.26 |
| 5 | 1352.82 | 1328.22 |
| 6 | 109.01 | 97.44 |
| 10 | 1102.23 | 1301.48 |
| 12 | 266.91 | 283.56 |
| 13 | 2114.55 | 2278.81 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r706566011
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
I did get partially blocked here because I was unable to use arrow arrays in Ballista's plans, and Ballista does use scalars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917882040
@yjshen thanks for offering the help. Really appreciate it! I have pushed my WIP commit to https://github.com/houqp/arrow-datafusion/tree/arrow2-merge. This branch merged with a relatively new commit 7932cb9373192ce2754b39c1f82f22c8a56b7266. I am still working to get it to build. My branch also requires a patched version of arrow2 at https://github.com/houqp/arrow2/tree/qp_ord (pending PR https://github.com/jorgecarleitao/arrow2/pull/398). I currently have arrow2 patched to point to my local check out in Cargo.toml.
My current goal is to get `cargo test --lib` to compile and run.
I think perhaps the best way for us to collaborate on this is to work on the [arrow2-merge](https://github.com/houqp/arrow-datafusion/tree/arrow2-merge) together. To reduce potential merge conflicts, I can start from the bottom of the build error output and you can start from the top. I have granted you write access to my datafusion fork, so you can push directly to the arrow2-merge branch if you want. Or you can send PRs to my branch if you want to get things reviewed. Once it's building again, I can squash the commit and push to this PR. I will try to push any work in progress commit to the arrow2-merge branch at the end of my day if you will have access to the latest fixes from my end when you wake up.
To set the right expectation, we might not be able to merge this PR when we have it all caught up to the latest master and get all tests to pass, etc. @jorgecarleitao still need to restart the process to donate arrow2 to ASF first before we can merge it to master. If you only need access to async parquet IO in the short term, @alamb 's comment in https://github.com/apache/arrow-rs/issues/111#issuecomment-917611415 would be a much faster route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-913018083
I am going to resume this PR and work on it on and off based on my available free time. Others are welcome to join me on the effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918020460
Thanks @houqp for the detailed explanation, I can follow the guidance and let me try to get involved.
> ..... I currently have arrow2 patched to point to my local check out in Cargo.toml.
Thanks, I can set up the project now.
> cargo test --lib to compile and run. I can start from the bottom of the build error output and you can start from the top.
OK, I'll start from the beginning.
> To set the right expectation, we might not be able to merge this PR when we have it all caught up to the latest master and get all tests to pass
I can make my own work rely on this branch instead. no worries on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918553501
FYI I was trying to run the TPC-H benchmarks in datafusion against the parquet dataset.
I had to add support for parquet (de)compression via datafusion:
```
-arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex"] }
+arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex", "io_parquet_compression"] }
```
And some small changes in the TPCH tool to make it compile.
The Parquet files generated by the benchmark were not readable yet though - this should be implemented here.
https://github.com/jorgecarleitao/arrow2/pull/402
Would be fun to have a full running benchmark against the same files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 676.98 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1241.67 | 1219.40 |
| 6 | 92.38 | 97.44 |
| 10 | 1057.55 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 1907.03 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 15% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-921527452
@Dandandan branch updated, @yjshen is a beast and knocked out all the build errors :D latest parquet2 main branch is required for building it locally, so you would need to patch that in cargo.toml.
We are now looking into the test failures. Once that's fixed, we will merge with master again to catch up on latest commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 725.57 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1241.67 | 1219.40 |
| 6 | 109.01 | 97.44 |
| 10 | 1102.23 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 2114.55 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 10% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r706565176
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
@jorgecarleitao do you remember what's reason for changing from vector of ScalarValue to Arrow array here? Is it for static typing of the individual list values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917578758
Hi all, I am here based on the need for async parquet read capability, as we already have object store API merged.
Since we will finally move to arrow2/parquet2, at the same time, it might be quite a lot of work for the current parquet to adapt for async reading. I suppose I shouldn't waste @alamb time on [this](https://github.com/apache/arrow-rs/pull/707).
I can devote non-trivial amounts of time to working on this PR, accelerating its maturity, and finally getting merged. @houqp could you share the progress currently and assign me anything that you think necessary? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are ~~18~~ ~~13~~ 12 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ei-grad commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
ei-grad commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-898582829
1. Could it be expected that arrow-rs would evolve in reasonable time to be on par with arrow2?
2. Could it worth a try to make a set of [Structural Search and Replace (SSR)](https://rust-analyzer.github.io/manual.html#structural-search-and-replace) patterns based on commits from this PR? Or is the kind of refactoring required too complex to achieve that way? (collect and apply a set of rules instead of continuously merging with ongoing development on the datafusion main branch)
3. Such set of patterns could be useful for arrow-rs to arrow2 migration in other projects too. Does it seem possible to make a universal arrow-rs->arrow2 migration tool utilizing SSR? Has anyone seen similar efforts for other libraries by a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1008810117
I'm working on a new integration branch between the arrow2 that still has RecordBatch and the latest datafusion merged into @houqp 's branch, porting decimal type support as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-847969377
A quick update here:
The benchmarks for TPCH 1 (the result itself is the same) are:
* CSV: 20% slower
* CSV reader is not parallel and does not re-use memory (working on this)
* group by uses `value(i)` which in arrow does not perform bound checks but arrow2 performs (@Dandandan 's change on the #339 should fix it).
* in-memory: 10% slower
* group by uses `value(i)` which in arrow does not perform bound checks but arrow2 performs (@Dandandan 's change on the #339 should fix it).
* without aggregation and in-memory: 2x faster
* arrow2's sorting is much faster
Arrow2 has a vectorized hash, but I have not used it yet.
With parquet, the first observation is that the conversion from CSV to parquet (i.e. `write`) is 10x faster:
master:
```
Converting './data/orders.tbl' to parquet files in directory './data/tpch-parquet/orders'
Conversion completed in 334242 ms
```
this PR:
```
Converting './data/orders.tbl' to parquet files in directory './data/tpch-parquet/orders'
Conversion completed in 33968 ms
```
The query itself is still 10% slower (even though reading is faster).
Investigations continue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-930034589
> But from the activities I have observed in the arrow2 repo so far
This is great information -- thank you for sharing it @houqp .
> so I am concerned that moving to ASF at its current stage would slow him down unless we start to get significant more contributions to the arrow2 repo.
Yeah - it is definitely a 2 edged sword -- I suspect moving arrow2 to ASF would indeed slow @jorgecarleitao down but also would encourage more contributions. Balancing that tradeoff is trickey for sure.
> If we don't want to pay the cost of maintaining a separate arrow2 datafusion branch, we could also temporarily have datafusion depend the current arrow2 repo with the goal of eventually merging arrow2 with arrow as it matures.
Since datafusion relies on arrow, if we change it to rely on arrow2, any downstream projects that rely on datafusion that want to use the latest datafusion will also have to change to use arrow2 (to feed data in the `MemoryProvider`, for example). I think this is what you mean by "downstream impact"
Thus I would prefer we continue datafusion with a stable implementation of arrow -- and we can wait until that describes arrow2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-985932549
Haven't gotten around to testing the io avro feature of the arrow2 crate and port the current reader to arrow2 dropping avro-rs, I'm certain it will greatly improve performance on it as well as half the time is spent reading avro and cloning its data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-977691372
we have now caught up to the the latest arrow2 and parquet2 version. next step would be pulling latest change from datafusion master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642561933
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
yeah, for me this was more like trying to get away from the `.value(i)`, which in arrow2 is very slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r707044869
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
For the protobuf ser/der, one thing we could do is to send the `dyan Array` over the wire as arrow ipc stream encoded as bytes in protobuf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-928852806
Hi all, thanks to @yjshen, we have reached a big milestone of passing all unit and integration tests with 0 cilppy error. There are couple minor issues need to be addressed, which are all tracked in the arrow2 github milestone. As for next steps, we will work on these minor issues while waiting for https://github.com/apache/arrow-datafusion/pull/1010 before we merge and catch up with master. After that, this PR should be ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922249010
Also comparing profiles reveals that the arrow2 spends more time / cycles in reading parquet data, e.g. for the first query about 43% of the time vs 37% in the master branch.
Some hot functions:
_$LT$parquet2..encoding..hybrid_rle..HybridRleDecoder$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::
arrow2::io::parquet::read::binary::basic::iter_to_array::
_$LT$arrow2..buffer..mutable..MutableBuffer$LT$A$GT$$u20$as$u20$core..iter..traits..collect..Extend$LT$A$GT$$GT$::extend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917882040
@yjshen thanks for offering the help. Really appreciate it! I have pushed my WIP commit to https://github.com/houqp/arrow-datafusion/tree/arrow2-merge. This branch merged with a relatively new commit 7932cb9373192ce2754b39c1f82f22c8a56b7266. I am still working to get it to build. My branch also requires a patched version of arrow2 at https://github.com/houqp/arrow2/tree/qp_ord (pending PR https://github.com/jorgecarleitao/arrow2/pull/398). I currently have arrow2 patched to point to my local check out in Cargo.toml.
My current goal is to get `cargo test --lib` to compile and run.
I think perhaps the best way for us to collaborate on this is to work on the [arrow2-merge](https://github.com/houqp/arrow-datafusion/tree/arrow2-merge) together. To reduce potential merge conflicts, I can start from the bottom of the build error output and you can start from the beginning. I have granted you write access to my datafusion fork, so you can push directly to the arrow2-merge branch if you want. Or you can send PRs to my branch if you want to get things reviewed. Once it's building again, I can squash the commit and push to this PR. I will try to push any work in progress commit to the arrow2-merge branch at the end of my day if you will have access to the latest fixes from my end when you wake up.
To set the right expectation, we might not be able to merge this PR when we have it all caught up to the latest master and get all tests to pass, etc. @jorgecarleitao still need to restart the process to donate arrow2 to ASF first before we can merge it to master. If you only need access to async parquet IO in the short term, @alamb 's comment in https://github.com/apache/arrow-rs/issues/111#issuecomment-917611415 would be a much faster route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-978951137
Running some new benchmarks - seems the recent changes in arrow2 have resulted in some really cool improvements (queries are about 20% faster now in this branch, some even more!). Will post some updated benchmarks later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-978951137
Running some new benchmarks - seems the recent changes in arrow2 have resulted in some really cool improvements (queries on parquet data are much faster now in this branch, some even more!). Will post some updated benchmarks later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642571782
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
In the hash join this `combine_hashes` function is used, which also could be used here:
```
fn combine_hashes(l: u64, r: u64) -> u64 {
let hash = (17 * 37u64).wrapping_add(l);
hash.wrapping_mul(37).wrapping_add(r)
}
```
e.g. something like: `*hash ^= combine_hashes(hash, other)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-913998272
@houqp , the argument `groups_filter: Arc<dyn Fn(usize, &RowGroupMetaData) -> bool>` of the `Reader` is used to filter row groups; the first argument is the row group, the second is its metadata, which includes all statistics.
In more recent versions we also have `Arc<dyn Fn(&PageHeader) -> bool>` to skip individual pages based on their statistics.
I am not familiar with this code-base, but I would expect that we create such mapping out of expressions: every row group is selected or not based on its metadata with statistics, and potentially its number in the parquet file for partitioned reads of a single file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r646083187
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
Thanks @Dandandan , this is now migrated; it uses the same that `hash_join` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-914019294
Works great, I will continue the migration later this week after it's merged. 55 more build errors to go, then I expect lots of test failures to troubleshoot :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-841776668
An update on this: most functionality has now been migrated. So far this results in a +1k -1.8k change.
There are some things missing:
* the `ScalarValue`, which imo currently has some hacks around it that we need to handle
* `prettyprint`, which is not in arrow2 yet.
* parquet and CSV read and write, as arrow2's has a different API to enable parallelism (e.g. see [the guide](https://jorgecarleitao.github.io/arrow2/io/parquet_read.html#parallelism-decoupling-of-cpu-from-io))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-921527452
@Dandandan branch updated, @yjshen is a beast and knocked all the build errors :D latest parquet2 main branch is required for building it locally, so you would need to patch that in cargo.toml.
We are now looking into the test failures. Once that's fixed, we will merge with master again to catch up on latest commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-929058068
@houqp and @yjshen and @jorgecarleitao -- what are your recommendations regarding converging arrow and arrow2?
arrow 6.0 is likely to be released a few weeks from now in mid october, so if we targeted arrow 7.0 (Jan / Feb 2022) as the timeline to converge that would let us get DataFusion over to arrow2 in a few weeks, and then have several months to work out the kinks prior to a real arrow 7.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-931202056
> Trying to figure out the constraints that we are working here: what does "stable" mean in this context? we accept breaking changes to the arrow crate between major versions, and thus the expectation of stability there is 3 months? Or do we expect arrow to not change significantly between major versions and thus have a longer expectation?
@jorgecarleitao
From my perspective some requirements for an arrow implementation used in DataFusion are:
1. The feature set is "close enough" to the current implementation of arrow that all DataFusion tests pass
2. There is a clear path, ability and willingness to help close any other gaps between the existing arrow-rs implementation
2. There is a community committed to its development and maintenance, under some sort of formal community project governance model (e.g. Apache Arrow)
In terms of technical requirements, I agree with @houqp that it is also desirable to be able to release bug fixes for some period of time that does not also require taking breaking API changes. However, this is largely mechanical work (eg. what I do for arrow-rs) and I think can be applied to most any codebase. What is needed is interest / willingness from the community to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007955176
> @alamb to answer your question, I think the arrow2 branch is ready to be tested more broadly now. I feel like we should merge this either to master or to an official arrow2 branch soon
I think making an official `arrow2` branch makes sense to me π
> without waiting for arrow2 1.x release, which is probably going to take a while. The API in arrow2 is still evolving fast right now (in a good direction), so I don't see it getting stablized any time soon. For example, it just recently removed the RecoardBatch abstraction entirely, which also aligns with our goal to come up with our own recordbatch abstraction in datafusion: #1248
I agree I see no reason to wait for a particular version of arrow2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1009708490
Ok I didn't see it because I was looking for a port of DecimalArray, my bad.
On Tue, Jan 11, 2022 at 2:57 AM QP Hou ***@***.***> wrote:
> @Igosuki <https://github.com/Igosuki>, @yjshen <https://github.com/yjshen>
> already ported decimal support into
> https://github.com/houqp/arrow-datafusion/tree/arrow2_merge. Thanks for
> helping with this, I will wait for your PR before sending a new PR to
> create our official arrow2 branch :)
>
> β
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1009529577>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADDFBVZPPYKZLKLDHCDRJLUVOFAPANCNFSM43RKTGGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp closed pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp closed pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-917611641
> Since we will finally move to arrow2/parquet2, at the same time, it might be quite a lot of work for the current parquet to adapt for async reading. I suppose I shouldn't waste @alamb time on this.
Moving datafusion to arrow2 definitely sounds like an approach to getting an async parquet reader. Depending on how trickey that proves, another approach might be to follow the path blazed by @jorgecarleitao in
https://github.com/jorgecarleitao/arrow2/pull/260 (that adds async parquet reading to arrow2) to add async reading to the existing parquet crate. I tried to write up some thoughts on https://github.com/apache/arrow-rs/issues/111#issue-867656782
> I can devote non-trivial amounts of time to working on this PR, accelerating its maturity, and finally getting merged. @houqp could you share the progress currently and assign me anything that you think necessary? Thanks!
That is awesome @yjshen ! Thank you again for all your help and contribution in this area
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r706565686
##########
File path: datafusion/src/scalar.rs
##########
@@ -64,7 +70,10 @@ pub enum ScalarValue {
/// large binary
LargeBinary(Option<Vec<u8>>),
/// list of nested ScalarValue
- List(Option<Vec<ScalarValue>>, DataType),
+ // 1st argument are the inner values (e.g. Int64Array)
+ // 2st argument is the Lists' datatype (i.e. it includes `Field`)
+ // to downcast inner values, use ListArray::<i32>::get_child()
+ List(Option<Arc<dyn Array>>, DataType),
Review comment:
`List[i] -> ScalarList` is a `O(1)` operation in arrow via `list.value(i).into()`, we make it a `O(N)` by casting each element to `scalar`.
Not married, though, was just trying to simplify the `scalar<->array` interaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 676.98 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1241.67 | 1219.40 |
| 6 | 92.38 | 97.44 |
| 10 | 1057.55 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 1907.03 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 15% slower, query 13 also takes a hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-924606019
Thanks to @yjshen , we have now caught up to 6402200c3ff1ca9732e85ca39fc89ae5f30cd965 :tada: .
Another potentially good news for @Dandandan 's benchmark work is I noticed I have accidentally reverted some of @jorgecarleitao 's hash aggregate changes while resolving conflicts for #808 . So we are working on getting those changes back, which might result in positive impact on performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-848444957
The differences in performance stems from the compute of the group by. Average time to read a 8k row batch in parquet during TPCH 1 on my computer:
* this PR: 500 us
* master: 1808 us
i.e. reading parquet is ~3.6x faster (single thread in both cases).
These improvements are not used atm because there is a bottleneck somewhere else (likely group by).
So, from the parquets' standpoint, this is imo a large improvement. I will now start touching the CPU, and remaining tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are ~~18~~ ~~13~~ ~~12~~ ~~10~~ 9 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are 19 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao closed pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao closed pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-921923054
I got one query running on a patched version of DataFusion / the benchmark on SF=10 / Parquet data source (split evenly in 16 partitions):
(output in ms, average of 3 runs )
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 6 | 883.02 | 1061.64 |
As you can see, this query runs a bit slower than the arrow-based version - didn't do any profiling or checking yet, but could be some primitives like hashing could be faster in the master branch right now.
If this is implemented
https://github.com/jorgecarleitao/arrow2/issues/418
I can probably get some more numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-851234013
```bash
cd datafusion && cargo test --lib
echo $?
0
```
i.e. the unit tests of datafusion tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1008810346
Will make a PR on @houqp 's fork when it's ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-979077578
Looking back at the earlier results I posted - it looks the main difference is that `arrow` / parquet got slower. I am not sure what the cause is.
Maybe it has to do with the extra validation @alamb has added to arrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-979928065
> Ah, it's exactly #1010 that introduces the regression.
FYI @rdettai we seem to have a regression related to the file ingest reordering. I wonder if it has something do with what files are read in what order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-841776668
An update on this: most functionality has now been migrated. So far this results in a +1k -1.8k change.
There are some things missing:
* the `ScalarValue`, which imo currently has some hacks around it that we need to handle
* `prettyprint`, which is not in arrow2 yet.
* parquet and CSV read and write, as arrow2's has a different API to enable parallelism (e.g. see [the guide](https://jorgecarleitao.github.io/arrow2/io/parquet_read.html#parallelism-decoupling-of-cpu-from-io))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1003995167
I merged the latest master and did some necessary updates to arrow2 here https://github.com/Igosuki/arrow-datafusion/tree/arrow22r (didn't fix handling timestamp logical types and decimals in arrow2).
I'm seeing significant performance improvements on an avro-related workload, and most of all datafusion barely appears in my flamegraph anymore.
Before

After

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004049147
oops, I think we've done something repetitive. @houqp and I also merged the latest master except for the Avro feature [here](https://github.com/houqp/arrow-datafusion/tree/arrow2_merge), I think we can merge both branches and get an includes-all branch here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004049147
oops, I think we've done something repetitive here. @houqp and I also merged the latest master except for the Avro feature [here](https://github.com/houqp/arrow-datafusion/tree/arrow2_merge), I think we can merge both branches and get an includes-all branch here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-844480669
```
cargo test --lib
...
Finished dev [unoptimized + debuginfo] target(s)
```
π
I temporarily deactivated the predicate pushdown for parquet, but since parquet2 is ~3-20x faster, I do not see it as a deal breaker. The `like` was also deactivated for a moment. The tests are not yet compiling; that is the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918020460
Thanks @houqp for the detailed explanation, I can follow the guidance and let me try to get involved.
> ..... I currently have arrow2 patched to point to my local check out in Cargo.toml.
Thanks, I can set up the project now.
> cargo test --lib to compile and run. I can start from the bottom of the build error output and you can start from the top.
OK, I'll start from the beginning.
> To set the right expectation, we might not be able to merge this PR when we have it all caught up to the latest master and get all tests to pass
I can make my own work rely on this branch instead. no worries on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922238494
Here the full table, thanks to @jorgecarleitao for implementing the parquet change so quickly.
(output in ms, average of 3 runs )
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 2071.80 | 2267.46 |
| 3 | 1615.33 | 1817.35 |
| 5 | 2990.24 | 3222.48 |
| 6 | 883.02 | 1061.64 |
| 8 | OOM | OOM |
| 9 | OOM | OOM |
| 10 | 2784.24 | 3096.37 |
| 12 | 1473.96 | 2039.88 |
| 13 | 6207.32 | 5965.40 |
We can see arrow is almost always faster than the arrow2 branch.
Now, preload the data into memory (skipping the parquet reader), it looks much more equal/mixed:
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 1775.07 | 1640.60 |
| 3 | 1128.12 | 1109.23 |
| 5 | 2528.62 | 2413.35 |
| 6 | 215.58 | 211.19 |
| 10 | 1886.62 | 2259.54 |
| 12 | 739.42 | 820.00 |
| 13 | 6397.59 | 6326.59 |
It seems some portion of the change could be that loading parquet is slower in arrow2, somehow.
E.g. loading the full lineitem table (16 files, 2.4GB total) takes ~8380 ms in arrow2 vs ~6484 ms in the master branch using the `-m` option in the benchmarking tool.
@jorgecarleitao any idea of why this might happen? I could do some profiling if that's useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922259322
This is really exciting work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-930954404
Trying to figure out the constraints that we are working here: what does "stable" mean in this context? we accept breaking changes to the `arrow` crate between major versions, and thus the expectation of stability there is 3 months? Or do we expect `arrow` to not change significantly between major versions and thus have a longer expectation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-928852806
Hi all, thanks to @yjshen, we have reached a big milestone of passing all unit and integration tests with 0 cilppy error. There are couple minor issues need to be addressed, which are all tracked in the arrow2 github milestone. As for next steps, we will work on these minor issues while waiting for https://github.com/apache/arrow-datafusion/pull/1010 before we merge and catch up with master. After that, this PR should be ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-932134550
> I think this PR has proven arrow2 has passed 1. For 2, I think arrow2 meets the requirement as well. For 3, we might need to give Jorge a bit more time to figure out the best governance model for the project and the community. Overall, I think things are looking great so far and I am seeing the end of the tunnel ;)
I agree 100% π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-914006491
> I am not familiar with this code-base, but I would expect that we create such mapping out of expressions: every row group is selected or not based on its metadata with statistics, and potentially its number in the parquet file for partitioned reads of a single file.
Our `build_row_group_predicate` function does exactly this right now. The issue I am trying to resolve is in order to build this mapping, I need access to metadata. So it looks like we will need to manually read the metadata first to create the mapping, then call `RecordReader::try_new` with the constructed `groups_filter` argument? This approach requires us to read the same metadata twice, once before calling `RecordReader::try_new`, once inside of `RecordReader::try_new`. Perhaps we could expose the metadata field from `RecordReader` and provide a `set_groups_filter` method to allow updating the row group filter after it's been created?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-979077578
Looking back at the earlier results I posted - it looks the main difference is that the original `arrow` / `parquet` got slower. I am not sure what the cause is.
Maybe it has to do with the extra validation @alamb has added to arrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
andygrove commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r675649915
##########
File path: ballista/rust/core/src/serde/physical_plan/from_proto.rs
##########
@@ -220,9 +229,36 @@ impl TryInto<Arc<dyn ExecutionPlan>> for &protobuf::PhysicalPlanNode {
proto_error("Unexpected empty window physical expression")
})?;
+<<<<<<< HEAD
Review comment:
@jorgecarleitao It looks like some merge conflicts got committed in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1010929254
https://github.com/houqp/arrow-datafusion/pull/17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-846506169
`cargo test --lib` runs now. There are ~~18~~ ~~13~~ ~~12~~ 10 failing tests to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642571782
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
In the hash join this `combine_hashes` function is used, which also could be used here as well:
```
fn combine_hashes(l: u64, r: u64) -> u64 {
let hash = (17 * 37u64).wrapping_add(l);
hash.wrapping_mul(37).wrapping_add(r)
}
```
e.g. something like: `*hash = combine_hashes(hash, other)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on a change in pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#discussion_r642571782
##########
File path: datafusion/src/physical_plan/hash_aggregate.rs
##########
@@ -339,6 +325,36 @@ pin_project! {
}
}
+fn hash_(group_values: &[ArrayRef]) -> Result<MutableBuffer<u64>> {
+ // compute the hashes
+ // todo: we should be able to use `MutableBuffer<u64>` to compute the hash and ^ them without
+ // allocating all the hashes before ^ them
+ let hashes = group_values
+ .iter()
+ .map(|x| {
+ let a = match x.data_type() {
+ DataType::Dictionary(_, d) => {
+ // todo: think about how to perform this more efficiently
+ // * first hash, then unpack
+ // * do not unpack at all, and instead figure out a way to leverage dictionary-encoded.
+ let unpacked = arrow2::compute::cast::cast(x.as_ref(), d)?;
+ arrow2::compute::hash::hash(unpacked.as_ref())
+ }
+ _ => arrow2::compute::hash::hash(x.as_ref()),
+ };
+ Ok(a?)
+ })
+ .collect::<Result<Vec<_>>>()?;
+ let hash = MutableBuffer::<u64>::from(hashes[0].values());
+
+ Ok(hashes.iter().skip(1).fold(hash, |mut acc, x| {
+ acc.iter_mut()
+ .zip(x.values().iter())
+ .for_each(|(hash, other)| *hash ^= other);
Review comment:
In the hash join this `combine_hashes` function is used, which also could be used here as well:
```
fn combine_hashes(l: u64, r: u64) -> u64 {
let hash = (17 * 37u64).wrapping_add(l);
hash.wrapping_mul(37).wrapping_add(r)
}
```
e.g. something like: `*hash ^= combine_hashes(hash, other)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-885992283
Currently, I have no time for this. If anyone else would like to take this, please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-914011654
Ahh, I see now. Thanks!
Would this work? https://github.com/jorgecarleitao/arrow2/pull/386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922212046
Ok, another round of update, @yjshen fixed all the tests while I am asleep :) I am going to merge with the latest master again this week to catch up on recent changes.
@Dandandan I have turned on the `io_parquet_compression` feature. You can push directly to jorge's arrow22 branch too if want. I can help manage the merge conflicts on my end if there is any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918853001
This is now included in arrow2 0.5.3 - let me know when this branch is updated, it seems to have a few incompatibilities
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-918553501
FYI I was trying to run the TPC-H benchmarks in datafusion against the parquet dataset.
I had to add support for parquet (de)compression via datafusion:
```
-arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex"] }
+arrow = { package = "arrow2", git = "https://github.com/jorgecarleitao/arrow2", rev = "43d8cf5c54805aa437a1c7ee48f80e90f07bc553", features = ["io_csv", "io_json", "io_parquet", "io_ipc", "io_print", "ahash", "merge_sort", "compute", "regex", "io_parquet_compression"] }
```
And some small changes in the TPCH tool to make it compile.
The Parquet files generated by the benchmark were not readable yet though - this is implemented here.
https://github.com/jorgecarleitao/arrow2/pull/402
Would be fun to have a full running benchmark against the same files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 725.57 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1241.67 | 1219.40 |
| 6 | 92.38 | 97.44 |
| 10 | 1102.23 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 2114.55 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 10% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
Running against in-memory data (to ignore parquet differences) based on https://github.com/jorgecarleitao/arrow2?rev=aa42ac252817a44097a4c503ac606b34ede342a0#aa42ac25
| Query | Arrow | Arrow2 |
|----------|:-------------:|------:|
| 1 | 725.57 | 615.13 |
| 3 | 507.22 | 489.79 |
| 5 | 1352.82 | 1219.40 |
| 6 | 109.01 | 97.44 |
| 10 | 1102.23 | 1247.18 |
| 12 | 266.91 | 279.14 |
| 13 | 2114.55 | 2082.04 |
Upgrading to latest arrow master seems to have improved some benchmarks quite a bit :tada: - I think most of it is due to the faster hashing kernel. Query of interest is q10, that is still around 10% slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-922307204
Currently my intuition is that the parquet files generated by the TPC-H convert + `parquet` are somehow less optimal for `parquet2` and `arrow2` than they are for`parquet`.
I tried to check this by also generating parquet files via this branch (which uses `parquet2`) but getting some errors when reading statistics (even when enabling statistics in the writer). The files generated are somewhat smaller (~10%) - so there is at least some difference.
I added the parquet (lineitem) files over here:
https://drive.google.com/drive/folders/19kUhg_6o8PuEOwCCLGQq0zh1bAq2TzG0?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-923695802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-930986589
> Trying to figure out the constraints that we are working here: what does "stable" mean in this context? we accept breaking changes to the arrow crate between major versions, and thus the expectation of stability there is 3 months?
Not sure about others, but that would be the case from my point of view. I even think 3 months is arbitrary since we could decouple our versioning from the main arrow version following the Julia implementation. So we can go even shorter than 3 months. But the main thing is avoiding breaking API change in minor and patch releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Igosuki commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
Igosuki commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004169207
I only did it to test out the latest versions of both arrow2 and datafusion.
There are breaking changes, and i had to make some small updates to arrow2.
Additionally, I didn't finish adding decimals and all time arrays are Int64
in tests which should be reverted eventually.
Regarding avro, as you can see on my branch the remaining code is quite
trivial so easy to copy and paste.
Le lun. 3 janv. 2022 Γ 13:43, Andrew Lamb ***@***.***> a
Γ©crit :
> @yjshen <https://github.com/yjshen> / @houqp <https://github.com/houqp> /
> @Igosuki <https://github.com/Igosuki> -- how close would you say your
> branches using arrow2 are ready to be used more broadly? Specifically, are
> you at the point where it would be possible to make a PR to datafusion to
> use the arrow2 crate?
>
> β
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004066438>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADDFBSXEIHIYIVSCMESVADUUGKVJANCNFSM43RKTGGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1004066438
@yjshen / @houqp / @Igosuki -- how close would you say your branches using arrow2 are ready to be used more broadly? Specifically, are you at the point where it would be possible to make a PR to datafusion to use the `arrow2` crate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #68: Experimenting with arrow2
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #68:
URL: https://github.com/apache/arrow-datafusion/pull/68#issuecomment-1007877042
Apologize for the late reply, still catching up to all the goodness happened during the break. I should have communicated this better to save @Igosuki some work :( But really impressive result, thank you for doing the benchmark!
I have squashed our latest round of merge fixes into a single commit to keep things clean and pushed it to this PR. What is missing right now from my point of view is migrating the python crate and avro. @Igosuki do you mind sending your avro changes to my branch at https://github.com/houqp/arrow-datafusion/tree/arrow2_merge so I can merge it into this PR? After that I think this PR is ready for a formal review with another round of quick master merge to pick up the removal of the python binding.
@alamb to answer your question, I think the arrow2 branch is ready to be tested more broadly now. I feel like we should merge this either to master or to an official arrow2 branch soon without waiting for arrow2 1.x release, which is probably going to take a while. The API in arrow2 is still evolving fast right now (in a good direction), so I don't see it getting stablized any time soon. For example, it just recently removed the RecoardBatch abstraction entirely, which also aligns with our goal to come up with our own recordbatch abstraction in datafusion: https://github.com/apache/arrow-datafusion/issues/1248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org