You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/05 02:24:16 UTC
[GitHub] [arrow-datafusion] andygrove opened a new pull request #263: WIP: Instructions for cross-compiling Ballista to the Raspberry Pi
andygrove opened a new pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263
# Which issue does this PR close?
Closes #247 .
# Rationale for this change
Raspberry Pi's are a fun and relatively inexpensive way to experiment with truly distributed computing.
# What changes are included in this PR?
Additional content for the user guide.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841824079
That is brutal 🤯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841823709
For the record, I am now able to run benchmarks against the 100GB data set on my Pi Kubernetes cluster:
```
$ microk8s.kubectl get pods
NAME READY STATUS RESTARTS AGE
ballista-scheduler-0 1/1 Running 1 17h
ballista-executor-5 1/1 Running 1 16h
ballista-executor-2 1/1 Running 1 16h
ballista-executor-3 1/1 Running 1 16h
ballista-executor-0 1/1 Running 5 17h
ballista-executor-4 1/1 Running 2 16h
ballista-executor-1 1/1 Running 1 17h
tpch 0/1 Completed 0 13m
```
## Results
```
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| A | F | 3775127758 | 5660776097194.455 | 5377736398183.935 | 5592847429515.929 | 25.499370423275426 | 38236.11698430493 | 0.050002243530928955 | 148047881 |
| N | F | 98553062 | 147771098385.98013 | 140384965965.03497 | 145999793032.7757 | 25.501556956882876 | 38237.19938880454 | 0.049985284338054006 | 3864590 |
| N | O | 7436302959 | 11150725648169.863 | 10593195276359.283 | 11016932215670.58 | 25.500009433521782 | 38237.227663621445 | 0.0499979183499098 | 291619616 |
| R | F | 3775724970 | 5661603032745.348 | 5378513563915.405 | 5593662252666.916 | 25.50006628406532 | 38236.697258453016 | 0.05000130433965413 | 148067261 |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
Query 1 avg time: 104835.61 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove edited a comment on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove edited a comment on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841823709
For the record, I am now able to run benchmarks against the 100GB data set on my Pi Kubernetes cluster:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: tpch
namespace: default
spec:
containers:
- image: andygrove/ballista-arm64
command: [ "/tpch",
"benchmark",
"--query=1",
"--path=/mnt/tpch/parquet-sf100-partitioned/",
"--format=parquet",
"--concurrency=24",
"--iterations=1",
"--debug",
"--host=ballista-scheduler",
"--port=50050"]
imagePullPolicy: Always
name: tpch
volumeMounts:
- mountPath: /mnt/tpch/parquet-sf100-partitioned/
name: data
restartPolicy: Never
volumes:
- name: data
persistentVolumeClaim:
claimName: data-pv-claim
```
```
$ microk8s.kubectl get pods
NAME READY STATUS RESTARTS AGE
ballista-scheduler-0 1/1 Running 1 17h
ballista-executor-5 1/1 Running 1 16h
ballista-executor-2 1/1 Running 1 16h
ballista-executor-3 1/1 Running 1 16h
ballista-executor-0 1/1 Running 5 17h
ballista-executor-4 1/1 Running 2 16h
ballista-executor-1 1/1 Running 1 17h
tpch 0/1 Completed 0 13m
```
## Results
```
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| A | F | 3775127758 | 5660776097194.455 | 5377736398183.935 | 5592847429515.929 | 25.499370423275426 | 38236.11698430493 | 0.050002243530928955 | 148047881 |
| N | F | 98553062 | 147771098385.98013 | 140384965965.03497 | 145999793032.7757 | 25.501556956882876 | 38237.19938880454 | 0.049985284338054006 | 3864590 |
| N | O | 7436302959 | 11150725648169.863 | 10593195276359.283 | 11016932215670.58 | 25.500009433521782 | 38237.227663621445 | 0.0499979183499098 | 291619616 |
| R | F | 3775724970 | 5661603032745.348 | 5378513563915.405 | 5593662252666.916 | 25.50006628406532 | 38236.697258453016 | 0.05000130433965413 | 148067261 |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
Query 1 avg time: 104835.61 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #263: WIP: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-832376857
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#263](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9091fd8) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/f7bd7b9f0e5a28b1fada2dbb60151b5af9b16545?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7bd7b9) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #263 +/- ##
=======================================
Coverage 76.81% 76.81%
=======================================
Files 133 133
Lines 23295 23295
=======================================
Hits 17895 17895
Misses 5400 5400
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f7bd7b9...9091fd8](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter edited a comment on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-832376857
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#263](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9af378) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/f7bd7b9f0e5a28b1fada2dbb60151b5af9b16545?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7bd7b9) will **decrease** coverage by `1.08%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #263 +/- ##
==========================================
- Coverage 76.81% 75.72% -1.09%
==========================================
Files 133 143 +10
Lines 23295 23840 +545
==========================================
+ Hits 17895 18054 +159
- Misses 5400 5786 +386
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [datafusion/src/physical\_plan/memory.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9tZW1vcnkucnM=) | `67.27% <0.00%> (-13.17%)` | :arrow_down: |
| [datafusion/src/physical\_plan/explain.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9leHBsYWluLnJz) | `51.35% <0.00%> (-6.23%)` | :arrow_down: |
| [datafusion/src/physical\_plan/empty.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9lbXB0eS5ycw==) | `83.82% <0.00%> (-5.73%)` | :arrow_down: |
| [datafusion/tests/custom\_sources.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi90ZXN0cy9jdXN0b21fc291cmNlcy5ycw==) | `71.87% <0.00%> (-5.18%)` | :arrow_down: |
| [benchmarks/src/bin/tpch.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVuY2htYXJrcy9zcmMvYmluL3RwY2gucnM=) | `30.82% <0.00%> (-4.54%)` | :arrow_down: |
| [datafusion/src/physical\_plan/limit.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9saW1pdC5ycw==) | `80.83% <0.00%> (-4.26%)` | :arrow_down: |
| [datafusion/src/error.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvZXJyb3IucnM=) | `51.85% <0.00%> (-3.71%)` | :arrow_down: |
| [datafusion/src/physical\_plan/coalesce\_batches.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9jb2FsZXNjZV9iYXRjaGVzLnJz) | `81.35% <0.00%> (-3.60%)` | :arrow_down: |
| [datafusion/src/physical\_plan/udaf.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi91ZGFmLnJz) | `71.42% <0.00%> (-3.58%)` | :arrow_down: |
| [datafusion/src/physical\_plan/expressions/mod.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9leHByZXNzaW9ucy9tb2QucnM=) | `67.85% <0.00%> (-3.58%)` | :arrow_down: |
| ... and [81 more](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f7bd7b9...d9af378](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter edited a comment on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-832376857
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#263](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9af378) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/f7bd7b9f0e5a28b1fada2dbb60151b5af9b16545?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7bd7b9) will **decrease** coverage by `1.08%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #263 +/- ##
==========================================
- Coverage 76.81% 75.72% -1.09%
==========================================
Files 133 143 +10
Lines 23295 23840 +545
==========================================
+ Hits 17895 18054 +159
- Misses 5400 5786 +386
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [datafusion/src/physical\_plan/memory.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9tZW1vcnkucnM=) | `67.27% <0.00%> (-13.17%)` | :arrow_down: |
| [datafusion/src/physical\_plan/explain.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9leHBsYWluLnJz) | `51.35% <0.00%> (-6.23%)` | :arrow_down: |
| [datafusion/src/physical\_plan/empty.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9lbXB0eS5ycw==) | `83.82% <0.00%> (-5.73%)` | :arrow_down: |
| [datafusion/tests/custom\_sources.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi90ZXN0cy9jdXN0b21fc291cmNlcy5ycw==) | `71.87% <0.00%> (-5.18%)` | :arrow_down: |
| [benchmarks/src/bin/tpch.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVuY2htYXJrcy9zcmMvYmluL3RwY2gucnM=) | `30.82% <0.00%> (-4.54%)` | :arrow_down: |
| [datafusion/src/physical\_plan/limit.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9saW1pdC5ycw==) | `80.83% <0.00%> (-4.26%)` | :arrow_down: |
| [datafusion/src/error.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvZXJyb3IucnM=) | `51.85% <0.00%> (-3.71%)` | :arrow_down: |
| [datafusion/src/physical\_plan/coalesce\_batches.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9jb2FsZXNjZV9iYXRjaGVzLnJz) | `81.35% <0.00%> (-3.60%)` | :arrow_down: |
| [datafusion/src/physical\_plan/udaf.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi91ZGFmLnJz) | `71.42% <0.00%> (-3.58%)` | :arrow_down: |
| [datafusion/src/physical\_plan/expressions/mod.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9leHByZXNzaW9ucy9tb2QucnM=) | `67.85% <0.00%> (-3.58%)` | :arrow_down: |
| ... and [81 more](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f7bd7b9...d9af378](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove merged pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove merged pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove merged pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove merged pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841824079
That is brutal 🤯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter edited a comment on pull request #263: WIP: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-832376857
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#263](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d7f4028) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/f7bd7b9f0e5a28b1fada2dbb60151b5af9b16545?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7bd7b9) will **decrease** coverage by `0.73%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #263 +/- ##
==========================================
- Coverage 76.81% 76.08% -0.74%
==========================================
Files 133 141 +8
Lines 23295 23734 +439
==========================================
+ Hits 17895 18057 +162
- Misses 5400 5677 +277
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [datafusion/src/physical\_plan/coalesce\_batches.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9jb2FsZXNjZV9iYXRjaGVzLnJz) | `83.18% <0.00%> (-1.77%)` | :arrow_down: |
| [datafusion/src/optimizer/hash\_build\_probe\_order.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvb3B0aW1pemVyL2hhc2hfYnVpbGRfcHJvYmVfb3JkZXIucnM=) | `62.06% <0.00%> (-0.54%)` | :arrow_down: |
| [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9kaXN0aW5jdF9leHByZXNzaW9ucy5ycw==) | `90.34% <0.00%> (-0.46%)` | :arrow_down: |
| [...ta/rust/core/src/serde/physical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9waHlzaWNhbF9wbGFuL2Zyb21fcHJvdG8ucnM=) | `47.34% <0.00%> (-0.22%)` | :arrow_down: |
| [...lista/rust/core/src/serde/logical\_plan/to\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9sb2dpY2FsX3BsYW4vdG9fcHJvdG8ucnM=) | `67.96% <0.00%> (-0.19%)` | :arrow_down: |
| [...ista/rust/core/src/serde/physical\_plan/to\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9waHlzaWNhbF9wbGFuL3RvX3Byb3RvLnJz) | `50.94% <0.00%> (-0.17%)` | :arrow_down: |
| [...sta/rust/core/src/serde/logical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9sb2dpY2FsX3BsYW4vZnJvbV9wcm90by5ycw==) | `40.45% <0.00%> (-0.11%)` | :arrow_down: |
| [datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9mdW5jdGlvbnMucnM=) | `89.42% <0.00%> (-0.06%)` | :arrow_down: |
| [datafusion-cli/src/main.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi1jbGkvc3JjL21haW4ucnM=) | `0.00% <0.00%> (ø)` | |
| [datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvbG9naWNhbF9wbGFuL3BsYW4ucnM=) | `82.01% <0.00%> (ø)` | |
| ... and [19 more](https://codecov.io/gh/apache/arrow-datafusion/pull/263/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f7bd7b9...d7f4028](https://codecov.io/gh/apache/arrow-datafusion/pull/263?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove edited a comment on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove edited a comment on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841823709
For the record, I am now able to run benchmarks against the 100GB data set on my Pi Kubernetes cluster:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: tpch
namespace: default
spec:
containers:
- image: andygrove/ballista-arm64
command: [ "/tpch",
"benchmark",
"--query=1",
"--path=/mnt/tpch/parquet-sf100-partitioned/",
"--format=parquet",
"--concurrency=24",
"--iterations=1",
"--debug",
"--host=ballista-scheduler",
"--port=50050"]
imagePullPolicy: Always
name: tpch
volumeMounts:
- mountPath: /mnt/tpch/parquet-sf100-partitioned/
name: data
restartPolicy: Never
volumes:
- name: data
persistentVolumeClaim:
claimName: data-pv-claim
```
```
$ microk8s.kubectl get pods
NAME READY STATUS RESTARTS AGE
ballista-scheduler-0 1/1 Running 1 17h
ballista-executor-5 1/1 Running 1 16h
ballista-executor-2 1/1 Running 1 16h
ballista-executor-3 1/1 Running 1 16h
ballista-executor-0 1/1 Running 5 17h
ballista-executor-4 1/1 Running 2 16h
ballista-executor-1 1/1 Running 1 17h
tpch 0/1 Completed 0 13m
```
## Results
```
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| A | F | 3775127758 | 5660776097194.455 | 5377736398183.935 | 5592847429515.929 | 25.499370423275426 | 38236.11698430493 | 0.050002243530928955 | 148047881 |
| N | F | 98553062 | 147771098385.98013 | 140384965965.03497 | 145999793032.7757 | 25.501556956882876 | 38237.19938880454 | 0.049985284338054006 | 3864590 |
| N | O | 7436302959 | 11150725648169.863 | 10593195276359.283 | 11016932215670.58 | 25.500009433521782 | 38237.227663621445 | 0.0499979183499098 | 291619616 |
| R | F | 3775724970 | 5661603032745.348 | 5378513563915.405 | 5593662252666.916 | 25.50006628406532 | 38236.697258453016 | 0.05000130433965413 | 148067261 |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
Query 1 avg time: 104835.61 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #263: Instructions for cross-compiling Ballista to the Raspberry Pi
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #263:
URL: https://github.com/apache/arrow-datafusion/pull/263#issuecomment-841823709
For the record, I am now able to run benchmarks against the 100GB data set on my Pi Kubernetes cluster:
```
$ microk8s.kubectl get pods
NAME READY STATUS RESTARTS AGE
ballista-scheduler-0 1/1 Running 1 17h
ballista-executor-5 1/1 Running 1 16h
ballista-executor-2 1/1 Running 1 16h
ballista-executor-3 1/1 Running 1 16h
ballista-executor-0 1/1 Running 5 17h
ballista-executor-4 1/1 Running 2 16h
ballista-executor-1 1/1 Running 1 17h
tpch 0/1 Completed 0 13m
```
## Results
```
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| A | F | 3775127758 | 5660776097194.455 | 5377736398183.935 | 5592847429515.929 | 25.499370423275426 | 38236.11698430493 | 0.050002243530928955 | 148047881 |
| N | F | 98553062 | 147771098385.98013 | 140384965965.03497 | 145999793032.7757 | 25.501556956882876 | 38237.19938880454 | 0.049985284338054006 | 3864590 |
| N | O | 7436302959 | 11150725648169.863 | 10593195276359.283 | 11016932215670.58 | 25.500009433521782 | 38237.227663621445 | 0.0499979183499098 | 291619616 |
| R | F | 3775724970 | 5661603032745.348 | 5378513563915.405 | 5593662252666.916 | 25.50006628406532 | 38236.697258453016 | 0.05000130433965413 | 148067261 |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
Query 1 avg time: 104835.61 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org