You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "dudzicp (via GitHub)" <gi...@apache.org> on 2023/02/14 10:11:18 UTC
[GitHub] [arrow-datafusion] dudzicp opened a new issue, #5276: Update benchmmarks on clickbench
dudzicp opened a new issue, #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276
In the [clickbench ](https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIChwYXJ0aXRpb25lZCkiOmZhbHNlLCJBdGhlbmEgKHNpbmdsZSkiOmZhbHNlLCJBdXJvcmEgZm9yIE15U1FMIjpmYWxzZSwiQXVyb3JhIGZvciBQb3N0Z3JlU1FMIjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiQ2l0dXMiOmZhbHNlLCJjbGlja2hvdXNlLWxvY2FsIChwYXJ0aXRpb25lZCkiOmZhbHNlLCJjbGlja2hvdXNlLWxvY2FsIChzaW5nbGUpIjpmYWxzZSwiQ2xpY2tIb3VzZSI6ZmFsc2UsIkNsaWNrSG91c2UgKHR1bmVkKSI6ZmFsc2UsIkNsaWNrSG91c2UgKHpzdGQpIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCI6ZmFsc2UsIkNyYXRlREIiOmZhbHNlLCJEYXRhYmVuZCI6ZmFsc2UsImRhdGFmdXNpb24iOnRydWUsIkFwYWNoZSBEb3JpcyI6ZmFsc2UsIkRydWlkIjpmYWxzZSwiRHVja0RCIChQYXJxdWV0KSI6dHJ1ZSwiRHVja0RCIjpmYWxzZSwiRWxhc3RpY3NlYXJjaCI6ZmFsc2UsIkVsYXN0aWNzZWFyY2ggKHR1bmVkKSI6ZmFsc2UsIkdyZWVucGx1bSI6ZmFsc2UsIkhlYXZ5QUkiOmZhbHNlLCJIeWRyYSI6ZmFsc2UsIkluZm9icmlnaHQiOmZhbHNlLCJLaW5ldGljYSI6ZmFsc2UsIk1hcmlhREIgQ29sdW1uU3RvcmUiOmZhbHNlLCJNYXJpYURCIjpmYWxzZSwiTW9uZXREQiI6ZmFsc2UsIk1vbmdvREIiOmZhbHNlLCJNeVNRTCAoTXlJU0FNKSI6ZmFsc2UsIk15U1FMIjpmYWxzZSwiUGlub3QiOmZh
bHNlLCJQb3N0Z3JlU1FMICh0dW5lZCkiOmZhbHNlLCJQb3N0Z3JlU1FMIjpmYWxzZSwiUXVlc3REQiAocGFydGl0aW9uZWQpIjpmYWxzZSwiUXVlc3REQiI6ZmFsc2UsIlJlZHNoaWZ0IjpmYWxzZSwiU2VsZWN0REIiOmZhbHNlLCJTaW5nbGVTdG9yZSI6ZmFsc2UsIlNub3dmbGFrZSI6ZmFsc2UsIlNRTGl0ZSI6ZmFsc2UsIlN0YXJSb2NrcyAodHVuZWQpIjpmYWxzZSwiU3RhclJvY2tzIjpmYWxzZSwiVGltZXNjYWxlREIgKGNvbXByZXNzaW9uKSI6ZmFsc2UsIlRpbWVzY2FsZURCIjpmYWxzZX0sInR5cGUiOnsic3RhdGVsZXNzIjp0cnVlLCJtYW5hZ2VkIjp0cnVlLCJKYXZhIjp0cnVlLCJjb2x1bW4tb3JpZW50ZWQiOnRydWUsIkMrKyI6dHJ1ZSwiTXlTUUwgY29tcGF0aWJsZSI6dHJ1ZSwicm93LW9yaWVudGVkIjp0cnVlLCJDIjp0cnVlLCJQb3N0Z3JlU1FMIGNvbXBhdGlibGUiOnRydWUsIkNsaWNrSG91c2UgZGVyaXZhdGl2ZSI6dHJ1ZSwiZW1iZWRkZWQiOnRydWUsIlJ1c3QiOnRydWUsInNlYXJjaCI6dHJ1ZSwiZG9jdW1lbnQiOnRydWUsInRpbWUtc2VyaWVzIjp0cnVlfSwibWFjaGluZSI6eyJzZXJ2ZXJsZXNzIjp0cnVlLCIxNmFjdSI6dHJ1ZSwiTCI6dHJ1ZSwiTSI6dHJ1ZSwiUyI6dHJ1ZSwiWFMiOnRydWUsImM2YS40eGxhcmdlLCA1MDBnYiBncDIiOnRydWUsImM1LjR4bGFyZ2UsIDUwMGdiIGdwMiI6dHJ1ZSwiYzZhLm1ldGFsLCA1MDBnYiBncDIiOnRydWUsIjE2IHRocmVhZHMiOnRydWUsIjIwIHRoc
mVhZHMiOnRydWUsIjI0IHRocmVhZHMiOnRydWUsIjI4IHRocmVhZHMiOnRydWUsIjMwIHRocmVhZHMiOnRydWUsIjQ4IHRocmVhZHMiOnRydWUsIjYwIHRocmVhZHMiOnRydWUsIm01ZC4yNHhsYXJnZSI6dHJ1ZSwiZjE2cyB2MiI6dHJ1ZSwiYzZhLjR4bGFyZ2UsIDE1MDBnYiBncDIiOnRydWUsInJhMy4xNnhsYXJnZSI6dHJ1ZSwicmEzLjR4bGFyZ2UiOnRydWUsInJhMy54bHBsdXMiOnRydWUsIlMyIjp0cnVlLCJTMjQiOnRydWUsIjJYTCI6dHJ1ZSwiM1hMIjp0cnVlLCI0WEwiOnRydWUsIlhMIjp0cnVlfSwiY2x1c3Rlcl9zaXplIjp7IjEiOnRydWUsIjIiOnRydWUsIjQiOnRydWUsIjgiOnRydWUsIjEyIjp0cnVlLCIxNiI6dHJ1ZSwiMzIiOnRydWUsIjY0Ijp0cnVlLCIxMjgiOnRydWUsInNlcnZlcmxlc3MiOnRydWUsInVuZGVmaW5lZCI6dHJ1ZX0sIm1ldHJpYyI6ImhvdCIsInF1ZXJpZXMiOlt0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlXX0=) Datafusion is currently ~5x slower than DuckDB on comparable machine. I think it would make sense to
update the benchmark results with the latest improvements so that datafusion will be a viable/reasonable option for new projects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1434756329
> The benchmark though should add parallelism by repartitioning after scanning - so I would expect aggregates to be executed in parallel (even though the scan won't be).
BTW When I ran
```
❯ SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
```
Locally using `datafusion-cli` it kept all my cores quite busy, so I don't think parallelism is the bottleneck for that particular query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on issue #5276: Update benchmmarks on clickbench
Posted by "jychen7 (via GitHub)" <gi...@apache.org>.
jychen7 commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1445276492
> The benchmark though should add parallelism by repartitioning after scanning - so I would expect aggregates to be executed in parallel (even though the scan won't be).
I recently look at Datafusion vs DuckDB for our use cases and one of query is a variant of ClickBench q23.
- Datafusion v19.rc1: 12s
- DuckDB v0.6.1: 0.6s (20x faster)
it aligns with "scan is not in parallel yet"
More detail here: https://github.com/apache/arrow-datafusion/issues/5404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #5276: Update benchmmarks on clickbench
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433639200
@alamb I'm running this on my machine now, not sure if its DISTINCT, once I have more details I'll file a ticket.
But anyway `COUNT(DISTINCT X)` is proven to be slow in most environments. If data is sorted by X you can jump between distinct values without scanning it, kinda skip scan. Otherwise people uses approximate distinct functions which are faster but has some inaccuracy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433655590
> I'll file a ticket if that possible to squeeze something from current distinct computation
Makes sense
Also in general this query looks like it is a test of how fast grouping can be done (in addition to the fact that our DISTINCT aggregator is pretty slow at the moment)
There are a bunch of ideas floating around about how to make Grouping faster -- for example https://github.com/apache/arrow-datafusion/issues/4973 and some details on https://github.com/apache/arrow-datafusion/issues/1570
I think someone just has to commit the time to hammering it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #5276: Update benchmmarks on clickbench
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1434487882
The benchmark though should add parallelism by repartitioning after scanning - so I would expect aggregates to be executed in parallel (even though the scan won't be).
Distinct count might be mostly slow as the current implementation is very allocation-heavy.
An additional suggestion to better performance on the benchmark is to add a target-cpu configuration which might yield a ~10% improvement - currently it compiles with the default target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] korowa commented on issue #5276: Update benchmmarks on clickbench
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1434253563
> One thing I noticed is that duckdb loads partitioned parquet files: https://github.com/ClickHouse/ClickBench/blob/main/duckdb-parquet/benchmark.sh as opposed to datafusion: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/benchmark.sh. I guess this can impact parallelism?
Not sure about DuckDB -- it seems to be able to parallelise reads of single file (at least latest version), but it definitely caused lack of parallelism for Datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1434753677
> Not sure about DuckDB -- it seems to be able to parallelise reads of single file (at least latest version), but it definitely caused lack of parallelism for Datafusion
@korowa s added the feature to parallelize reads of a single parquet file in DataFusion. https://github.com/apache/arrow-datafusion/pull/5295 proposes enabling this by default
> Distinct count might be mostly slow as the current implementation is very allocation-heavy.
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on issue #5276: Update benchmmarks on clickbench
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1430417955
I think this is something we should do as soon as possible. We should not needlessly create a false impression of "slow" in the eyes of the community at large. Why try to correct it later when we can avoid it?
Our benchmark was not run on a comparable machine, and it is a very old version. I suggest we run it on `c6a.metal, 500gb gp2` (that's the machine the majority is gravitating towards) with our current version. After that, we can identify which queries we are slow at, investigate the reasons and improve.
We (the Synnada team) is happy to help with investigations and take on improvement tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1435673155
> I guess @alamb is correct and grouping/aggregation code needs improvements.
This is one of the last areas I think there are factors of 2 or more performance to be extracted given the current architecture 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on issue #5276: Update benchmmarks on clickbench
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1431857865
@waitingkuo, if you create the issues that identify what is required to run the queries properly, we can help with addressing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1441917340
> ClickBench is almost for grouping/aggregation and high cardinality distinct, so If datafusion wants to have better performance in this bench, it must need a high-performance hashtable which suits for OLAP queries.
Yes; I am not sure I would necessarily say it was the hashtable itself that needs work, more like how the values in the hashtable are used / updated - but that is likely being pedantic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #5276: Update benchmmarks on clickbench
Posted by "waitingkuo (via GitHub)" <gi...@apache.org>.
waitingkuo commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1431892334
@ozankabak thank you, i'll submit another ticket to do so.
I just submit the pr to update it to v18
https://github.com/ClickHouse/ClickBench/pull/77
here's the summary:
1 of the query get 20 times boost
1 of the query remain 0.1 times performance
1 of the query still not working
except from these 3 outliers, we have 20% improvement in average
the histogram chart
x axis means: ```old execution time / new execution time``` the larger the more improvement, 1.0 means no change
<img width="418" alt="image" src="https://user-images.githubusercontent.com/1100923/219127094-05c45364-15b0-4a5c-8de7-c7cb0133b165.png">
here's the result query by query
```bash
Query 0: 0.9776802049030369
Query 1: 1.707491082045184
Query 2: 1.2640586797066016
Query 3: 0.9203807875378623
Query 4: 0.9241755388210855
Query 5: 0.94172723106135
Query 6: 1.649497487437186
Query 7: 1.8705738705738708
Query 8: 0.9532640949554897
Query 9: 0.09891081294396213
Query 10: 0.9790727043117446
Query 11: 1.0668953687821612
Query 12: 1.0256627922162727
Query 13: 0.9520421488719772
Query 14: 1.0535746389404925
Query 15: 1.1839474435875463
Query 16: 1.2163045644807655
Query 17: 1.0196377825847225
Query 18: 1.2491737143968116
Query 19: 0.9741868869385649
Query 20: 1.2916241203256522
Query 21: 0.9167798624671026
Query 22: 0.9743192470077857
Query 23: 1.0809550243337944
Query 24: 1.0715070085911764
Query 25: 1.868698910081744
Query 26: 1.8450380677343134
Query 27: 0.8860306599462013
Query 28: 18.859884645982497
Query 29: 2.9667709147771695
Query 30: 1.0043050430504306
Query 31: 0.9922544851934578
Query 32: doesn't work
Query 33: 1.2395144767868875
Query 34: 1.2316816529558063
Query 35: 1.1153739994479712
Query 36: 0.9655502392344499
Query 37: 0.9329896907216495
Query 38: 0.979089790897909
Query 39: 1.0419161676646707
Query 40: 0.9942028985507246
Query 41: 1.028301886792453
Query 42: 1.039426523297491
```
outlier queries:
Query 9: 0.09891081294396213 -> 10 times worse
```sql
SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
```
i have no clue for now, perhaps due to `DISTINCT`
@alamb @tustvold @comphead do you have any idea for this?
Query 28: 18.859884645982497 -> almost 20 times better
```sql
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
```
i think this is benifit from the improvement of `regexp_replace`
Query 32: query doesn't work
```sql
SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;
```
when i run it, it's killed (same as the previous version)
```bash
willy@willybench:~/ClickBench/datafusion$ datafusion-cli -f create.sql q33.sql
DataFusion CLI v18.0.0
0 rows in set. Query took 0.039 seconds.
Killed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433617359
> Perhaps we could rename datafusion to datafusion(parquet) so that other people could easily understanding what they're comparing. (btw, we can judge it by loading time and datasize as well)
I think this would be very helpful. Yes I think we should be only comparing datafusion performance to parquet files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #5276: Update benchmmarks on clickbench
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1430559129
I wasn't really too involved with this. I have been focussing more on TPC-H and TPC-DS, and I also see DuckDB performing about 5x faster than DataFusion. I have not spent time determining why this is the case yet.
The latest results are here. This is yet another half-finished project of mine, so I apologize for the lack of configuration info. I am mostly running with default configs.
https://sqlbenchmarks.io/sqlbench-h/results/env/workstation/sf10/single_node/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] kmitchener commented on issue #5276: Update benchmmarks on clickbench
Posted by "kmitchener (via GitHub)" <gi...@apache.org>.
kmitchener commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1438624668
I've re-run the benchmark on a c6a.4xlarge instance similar to the other stateless engines in the benchmark. Submitted it here -> https://github.com/ClickHouse/ClickBench/pull/81 . For nearly all requests, it performs significantly better, but I wonder if that's due in large part to the `datafusion.optimizer.repartitioning_file_scans` by default that was committed yesterday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jhorstmann commented on issue #5276: Update benchmmarks on clickbench
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433171016
AFAIK one big difference regarding performance is that most other engines in the clickbench suite first load the data into memory or some other internal format, while datafusion scans directly from the parquet files. Extending the benchmark runner to support both use cases and report separate results would be very useful. [The tcph benchmark runner for example has such a feature](https://github.com/apache/arrow-datafusion/blob/main/benchmarks/src/bin/tpch.rs#L96)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1441699984
> but I wonder if that's due in large part to the datafusion.optimizer.repartitioning_file_scans by default that was committed yesterday.
It would not at all surprise me
Thank you @kmitchener
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on issue #5276: Update benchmmarks on clickbench
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1432049279
@waitingkuo, I have a few quick questions: (1) Which machine did your new results run on, is it the same machine? (2) Which features are missing to run the queries as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #5276: Update benchmmarks on clickbench
Posted by "waitingkuo (via GitHub)" <gi...@apache.org>.
waitingkuo commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1431466976
the codes to reproduce the benchmark are here https://github.com/ClickHouse/ClickBench/tree/main/datafusion
The way to update the data in the clickbench website is to update the json file in the results folder and send a PR https://github.com/ClickHouse/ClickBench/tree/main/datafusion/results
The current result listed in the website was benchmarked by datafusion v11, which was release around 6 months ago.
to imporve:
1. use datafusion v18 to rerun the benchmark codes and send the PR. t
2. at the time I wrote the benchmark codes, datafusion didnt support some features. e.g. it didn't support schema from parquet, so for some data type like timestamp, we need to load it as string and then cast it to timestamp explicitly. e.g.
this is the original sql queries
```sql
SELECT URLHash, EventDate, COUNT(*) AS PageViews FROM ...
```
https://github.com/ClickHouse/ClickBench/blob/main/duckdb-parquet/queries.sql#L41
this is the modified version for datafusion
```sql
SELECT "URLHash", "EventDate"::INT::DATE, COUNT(*) AS PageViews FROM
```
https://github.com/ClickHouse/ClickBench/blob/main/datafusion/queries.sql#L41
I did modify some quries so that it works in datafusion. To fix this, we need to verify whether the new datafusion work or not. If so we could update the quries. If not, we could fire the issue to improve
I'll do the first option soon (update to v18), it should be a quick improvement.
It'll take more time to do the second approach. Welcome for the contribution. I'll get back to this if there's no one work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on issue #5276: Update benchmarks on clickbench
Posted by "jychen7 (via GitHub)" <gi...@apache.org>.
jychen7 commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1504099883
I just run a benchmark on datafusion v22: https://github.com/ClickHouse/ClickBench/pull/97
Glad that https://github.com/apache/arrow-datafusion/issues/5325 fix the regression on `q9`
```
SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
```
|| v11 | v18 | v22 |
|-|-|-|-|
|q9|5s|52s|3.5s|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #5276: Update benchmarks on clickbench
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1504554000
@jychen7 glad to hear #5325 fixed
for #5969 are you running with artifact built in `release` mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1463745359
Filed https://github.com/apache/arrow-datafusion/issues/5547 to track improving COUNT DISTINCT query performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #5276: Update benchmmarks on clickbench
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1432033762
> Query 9: 0.09891081294396213 -> 10 times worse
>
> SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
> i have no clue for now, perhaps due to DISTINCT
> @alamb @tustvold @comphead do you have any idea for this?
@waitingkuo I can try to experiment if this is due to distinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #5276: Update benchmmarks on clickbench
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1435078254
> The last suggestion might be better for running in a non-distributed execution, the current rule is better optimized for distributed execution (as shuffling data might be expensive), so if we get some improvements there, we'll need to make that configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] dudzicp commented on issue #5276: Update benchmmarks on clickbench
Posted by "dudzicp (via GitHub)" <gi...@apache.org>.
dudzicp commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433678039
One thing I noticed is that duckdb loads partitioned parquet files: https://github.com/ClickHouse/ClickBench/blob/main/duckdb-parquet/benchmark.sh as opposed to datafusion: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/benchmark.sh. I guess this can impact parallelism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433619317
> @waitingkuo I can try to experiment if this is due to distinct.
@comphead thank you -- it would be super helpful if you could open a ticket with your analysis (if you haven't already)
I think doing something like posting the EXPLAIN plan and the output of `EXPLAIN ANALYZE` would be very helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #5276: Update benchmmarks on clickbench
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1434991533
@dudzicp
The repartitioning shouldn't be on grouping key, as hash repartitioning itself is CPU intensive. So for optimal concurrency, you want that to be parallelized as well. The round-robin partitioning is very cheap to do in DataFusion.
In certain cases it might be beneficial to do hash repartition before the (partial)`AggregateExec` and only perform a single aggregation, now we're basically performing the work twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] dudzicp commented on issue #5276: Update benchmmarks on clickbench
Posted by "dudzicp (via GitHub)" <gi...@apache.org>.
dudzicp commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1431924580
With results from @andygrove being consistent with clickbench (~5x difference compared to duckdb), I think it make more sense to focus more on profiling to narrow the gap, rather than re running the benchmarks. Also when running the benchmarks, make sure you enable important config options mentioned by @korowa: https://github.com/apache/arrow-datafusion/discussions/5142#discussioncomment-4880005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #5276: Update benchmmarks on clickbench
Posted by "waitingkuo (via GitHub)" <gi...@apache.org>.
waitingkuo commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1432056535
@ozankabak i used azure's Standard_F16s_v2 vm to do so which has 16 vcpu and 32GB memory
https://learn.microsoft.com/en-us/azure/virtual-machines/fsv2-series
i didnt looks into the issue that the process killed yet. my quick guess is that it runs out of memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1653389613
I filed https://github.com/apache/arrow-datafusion/issues/7108 to track updating the clickbench benchmarks again, let's continue the conversation there. Closing this ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #5276: Update benchmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5276: Update benchmarks on clickbench
URL: https://github.com/apache/arrow-datafusion/issues/5276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1430501398
I agree this would be great to do. @waitingkuo and @andygrove have worked on this in the past and perhaps they have thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #5276: Update benchmmarks on clickbench
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433650162
I'll file a ticket if that possible to squeeze something from current distinct computation
```
❯ SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
+----------+-----------------------+----------+---------------------------+-----------------------------+
| RegionID | SUM(hits.AdvEngineID) | c | AVG(hits.ResolutionWidth) | COUNT(DISTINCT hits.UserID) |
+----------+-----------------------+----------+---------------------------+-----------------------------+
| 229 | 2077656 | 18295832 | 1506.085243130785 | 2845673 |
| 2 | 441662 | 6687587 | 1479.8386542111527 | 1081016 |
| 208 | 285925 | 4261812 | 1285.2593246722286 | 831676 |
| 169 | 100887 | 3320229 | 1465.9073732564832 | 604583 |
| 32 | 81498 | 1843518 | 1538.0376568061718 | 216010 |
| 34 | 161779 | 1792369 | 1548.360152401654 | 299479 |
| 184 | 55526 | 1755192 | 1506.8082967561384 | 322661 |
| 42 | 108820 | 1542717 | 1587.1085208758313 | 243181 |
| 107 | 120470 | 1516690 | 1548.6028970982863 | 272448 |
| 51 | 98212 | 1435578 | 1579.8860354505293 | 211505 |
+----------+-----------------------+----------+---------------------------+-----------------------------+
10 rows in set. Query took 559.109 seconds.
❯ SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
+----------+-----------------------+----------+---------------------------+
| RegionID | SUM(hits.AdvEngineID) | c | AVG(hits.ResolutionWidth) |
+----------+-----------------------+----------+---------------------------+
| 229 | 2077656 | 18295832 | 1506.085243130785 |
| 2 | 441662 | 6687587 | 1479.8386542111527 |
| 208 | 285925 | 4261812 | 1285.2593246722286 |
| 169 | 100887 | 3320229 | 1465.9073732564832 |
| 32 | 81498 | 1843518 | 1538.0376568061718 |
| 34 | 161779 | 1792369 | 1548.360152401654 |
| 184 | 55526 | 1755192 | 1506.8082967561384 |
| 42 | 108820 | 1542717 | 1587.1085208758313 |
| 107 | 120470 | 1516690 | 1548.6028970982863 |
| 51 | 98212 | 1435578 | 1579.8860354505293 |
+----------+-----------------------+----------+---------------------------+
10 rows in set. Query took 32.514 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] sundy-li commented on issue #5276: Update benchmmarks on clickbench
Posted by "sundy-li (via GitHub)" <gi...@apache.org>.
sundy-li commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1441911393
> I guess @alamb is correct and grouping/aggregation code needs improvements.
That's right, duckdb has very impressive grouping/aggregation improvements. Details in [blog](https://duckdb.org/2022/03/07/aggregate-hashtable.html).
ClickBench is almost for `grouping/aggregation` and `high cardinality distinct`, so If datafusion wants to have better performance in this bench, it must need a high-performance hashtable which suits for OLAP queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on issue #5276: Update benchmarks on clickbench
Posted by "jychen7 (via GitHub)" <gi...@apache.org>.
jychen7 commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1504223131
btw, glad that https://github.com/apache/arrow-datafusion/issues/5325 fix the regression on `q9`
```
SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
```
I also create an issue for q32: #5969
```
SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10
```
Summary of queries that is 2x slower than DuckDB
|ClickBench| Datafusion v18 (parquet) | Datafusion v22 (parquet) | DuckDB 2022-11 (parquet)| Issue Tracker |
|-|-|-|-|-|
|q0|0.22s|0.22s|0.03s|
|q4|2.66s|2.7s|0.79s|
|q8|2.87s|3s|0.9s|
|q9|52s|3.5s| 1.29s | #5325 |
|q12|2s|2s|0.76s|
|q13|3.62s|3.7s|1.21s|
|q14|2.08s|2.08s|0.84s|
|q15|3.42s|0.84s|
|q16|4.99s|5.0s|1.86s|
|q17|4.32s|4.34|1.81s|
|q18|10.74s|10.9s|3.49s|
|q30|2.88s|2.94s|0.92s|
|q31|5.22s|4.4s|1.14s|
|q32|-|-|5.28s| #5969 |
|q33|8.54s|8.7s|3.15s|
|q34|9.01s|9.2s|3.19s|
|q35|3.66s|3.5s|0.88s|
|q36|0.4s|0.4s|0.18s|
|q39|0.77s|0.77s|0.32s|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #5276: Update benchmmarks on clickbench
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1435077923
The last suggestion might be better for running in a non-distributed, the current rule is better optimized for distributed execution (as shuffling data might be expensive), so if we get some improvements there, we'll need to make that configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #5276: Update benchmmarks on clickbench
Posted by "waitingkuo (via GitHub)" <gi...@apache.org>.
waitingkuo commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1433231387
@jhorstmann thank you for pointing out this
some system like duckdb or clickbench does provide two kinds of setting. e.g.
<img width="794" alt="image" src="https://user-images.githubusercontent.com/1100923/219401169-d4066321-d079-4207-8640-3a004a58f325.png">
the original comparison @dudzicp posted is to compare duckdb(parquet) with datafusion. I think it's fair enough. Perhaps we could rename datafusion to datafusion(parquet) so that other people could easily understanding what they're comparing. (btw, we can judge it by loading time and datasize as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #5276: Update benchmmarks on clickbench
Posted by "waitingkuo (via GitHub)" <gi...@apache.org>.
waitingkuo commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1432070491
@comphead thank you
the quickest way is pull clickbench github repo and go tho this folder
https://github.com/ClickHouse/ClickBench/tree/main/datafusion
and then
```bash
# Download benchmark target data
wget --continue https://datasets.clickhouse.com/hits_compatible/hits.parquet
# launch datafusion-cli
datafusion-cli
DataFusion CLI v18.0.0
> CREATE EXTERNAL TABLE hits STORED AS PARQUET LOCATION 'hits.parquet';
> SELECT "RegionID", SUM("AdvEngineID"), COUNT(*) AS c, AVG("ResolutionWidth"), COUNT(DISTINCT "UserID") FROM hits GROUP BY "RegionID" ORDER BY c DESC LIMIT 10;
```
to run the full benchmark, you could do ```bash benchmark.sh``` which contains
1. install latest datafusion
2. download benchmark dataset
3. execute 3 times per query
it take around 20-30 minutes for me for a full benchmark generation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on issue #5276: Update benchmarks on clickbench
Posted by "jychen7 (via GitHub)" <gi...@apache.org>.
jychen7 commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1504574207
> for https://github.com/apache/arrow-datafusion/issues/5969 are you running with artifact built in release mode?
yes, it is release mode. For detail, can refer to 'benchmark.sh' in https://github.com/ClickHouse/ClickBench/pull/97/files.
btw, q32 not working is a known issue. I didn't find related github issue, so create #5969 to track it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5276: Update benchmarks on clickbench
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5276:
URL: https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1633094035
I expect DataFusion 28 to perform much better (and q32 now works) due to https://github.com/apache/arrow-datafusion/pull/6904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org