You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "waynexia (via GitHub)" <gi...@apache.org> on 2026/01/15 09:14:22 UTC
[I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
waynexia opened a new issue, #19826:
URL: https://github.com/apache/datafusion/issues/19826
### Is your feature request related to a problem or challenge?
Follow on to
- https://github.com/apache/datafusion/issues/17721
- https://github.com/apache/datafusion/issues/16643
- #14587
- #18909
### Describe the solution you'd like
Now that DataFusion 52.0.0 is released, It would be great to update ClickBench https://benchmark.clickhouse.com/ with the newest version and highlight and improvement. The current result is still of version 47.0.0
ClickBench is a benchmark heavy on filtering and aggregation that we have used as an optimization target for the last several releases.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3759739443
> For 4xlarge instances, almost all cases are improved, but I observed some regression in small instances (2x or x), like q3 q4 q5 now needs 2~3 seconds to run ([here](https://github.com/ClickHouse/ClickBench/pull/749/changes#diff-6887ea4cb78ed75a19622030189b040882bab594e1ad5587052dcc46e67919e5R16-R18)).
@waynexia I suspect this is a result of using more memory somehow (and this swapping more).
What I think need to happen is that DataFusion's "out of core" story needs to improve (aka we shouldn't need swap space, we should be using the memory manager and spill mechanism).
However, when I last tried this I encountered issues running some of the queries
- https://github.com/apache/datafusion/issues/19425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3976913221
I made an alternate proposal here
- https://github.com/ClickHouse/ClickBench/pull/804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3755164374
I am not familiar enough with clickbench to have an opinion on what 'update the results as-is' means. I believe @pmcgleenon was the last person to update the clickbench results, perhaps he would be able to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-4011917927
A sneak peek at the results
New results (1.45 away from the 🦆 )
<img width="875" height="984" alt="Image" src="https://github.com/user-attachments/assets/655e8c39-2b31-4b27-b00c-0cccb5e6921c" />
[Current results](https://benchmark.clickhouse.com/#system=+curp|ti%20rud|kBP%20t&type=-&machine=+ca4e&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-) (1.61 away from the 🦆 ):
<img width="856" height="975" alt="Image" src="https://github.com/user-attachments/assets/80265fef-0cdc-4d13-b8b7-9c9239de1cd5" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3758685717
>I think we should remove c6a.xlarge from the Clickbench results. It can be added back in when the underlying issues get resolved
+1. On the other hand, `c6a.xlarge` takes lots of time to run (compile + bench). For what I got this time, `c6a.xlarge` and `c6a.2xlarge` are reflecting a similar trend, keeping only `c6a.2xlarge` and above looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3759741876
So what I think we should do is
1. "publish the numbers"
2. Then work in parallel on improving performance for the "can't fit everything into RAM" case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3753896697
I've updated both `datafusion` and `datafusion-partitioned` for four types of instances, PR https://github.com/ClickHouse/ClickBench/pull/749.
For 4xlarge instances, almost all cases are improved, but I observed some regression in small instances (2x or x), like q3 q4 q5 now needs 2~3 seconds to run ([here](https://github.com/ClickHouse/ClickBench/pull/749/changes#diff-6887ea4cb78ed75a19622030189b040882bab594e1ad5587052dcc46e67919e5R16-R18)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3789350493
Benchmark script is updated to use a prebuilt binary. https://github.com/ClickHouse/ClickBench/pull/749 looks ready for review. What do you think about it @alamb @Omega359 @pmcgleenon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-4011698043
I have a new PR to update to 52
- https://github.com/ClickHouse/ClickBench/pull/810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3759742548
Thank you all for all your work on this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3758565971
> It turns out that I forgot to disable swap for queries... q18 ([here](https://github.com/ClickHouse/ClickBench/pull/749/changes#diff-c716a873880f5c1ed29736e4bd16df7a8985fcf94d64a6994c8abc014f7d7b2fR31)) will be killed without swap memory 🥲
That's the danger of using swap. From what I understand swap is needed to compile datafusion-cli on the [`c6a.xlarge`](https://instances.vantage.sh/aws/ec2/c6a.xlarge) instance type which only has 8GB of RAM. I think we should remove `c6a.xlarge` from the Clickbench results. It can be added back in when the underlying issues get resolved
Any thoughts @alamb @Omega359 @waynexia ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3753922586
I prefer to update those results as-is, as at least all cases can run on every instance (xlarge has swap space), and the current result is a bit outdated. What do you think? cc @alamb @Omega359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3790597958
Just need to update the title for the PR and I think you're good to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3759715257
> I prefer to update those results as-is, as at least all cases can run on every instance (xlarge has swap space), and the current result is a bit outdated. What do you think? cc @alamb @Omega359
I agree we should update the results to reflect the current state of DataFusion. If the results are poor or regressed on the smaller instances, I think it is fine to post those results here (and then we can file tickets to improve the results in subsequent runs)
> That's the danger of using swap. From what I understand swap is needed to compile datafusion-cli on the [c6a.xlarge](https://instances.vantage.sh/aws/ec2/c6a.xlarge) instance type which only has 8GB of RAM. I think we should remove c6a.xlarge from the Clickbench results. It can be added back in when the underlying issues get resolved
I personally think we should leave them in (it will make it clearer where we need to work to improve)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3757997907
>I didn't see any null results in the c6a.xlarge this time, did we already make some improvements to run on a low memory system?
I'm not very sure if I disabled swap (for building datafusion) before running queries 🙈 re-running on xlarge to double check...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "rschu1ze (via GitHub)" <gi...@apache.org>.
rschu1ze commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3765471837
So swap is needed to compile DataFusion and not for the queries itself? Should be fine to enable it temporarily. I just wonder why you guys don't publish binaries or a Docker image somewhere (GitHub, homepage, DockerHub) and download/run these in the ClickBench scipts.
Side note: swapping would actually also be fine in queries if it is needed to make queries go through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #19826: Update ClickBench benchmarks with DataFusion 52.0.0
URL: https://github.com/apache/datafusion/issues/19826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-4023002558
https://github.com/ClickHouse/ClickBench/pull/810 is merged and our results are live:
https://benchmark.clickhouse.com/#system=+curp|ti%20rud|kBP%20t&type=-&machine=+ca4e&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3758159278
It turns out that I forgot to disable swap for queries... q18 ([here](https://github.com/ClickHouse/ClickBench/pull/749/changes#diff-c716a873880f5c1ed29736e4bd16df7a8985fcf94d64a6994c8abc014f7d7b2fR31)) will be killed without swap memory 🥲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3765501351
> I just wonder why you guys don't publish binaries or a Docker image somewhere (GitHub, homepage, DockerHub) and download/run these in the ClickBench scipts.
Mostly b/c datafusion-cli isn't really a product (it is more like a testing tool) so we don't typically have users who are not also developers. I think we also historically wanted to compile datafusion-cli targeting the appropriate hardware (e.g. to use the AVX-512 instructions if available)
There is no reason we can't publish binaries though
🤔 One thing we could try is use the prebuilt binaries for `datafusion-cli` from homebrew If the performance is the same:
https://docs.brew.sh/Homebrew-on-Linux
https://formulae.brew.sh/formula/datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3760424775
> Should we publish the xlarge case number with swap enabled (which is the current number in [ClickHouse/ClickBench#749](https://github.com/ClickHouse/ClickBench/pull/749)), or rerun with swap disabled? I still kept the instance and compiled artifact, so I can get one soon. I'm wondering how it was done previously. This time it would be oom-killed on those queries, and the benchmark script is interrupted. Is it done manually?
I am not sure how it was done previously. Maybe @pmcgleenon or @rschu1ze knows (I think @rschu1ze ran it last time maybe)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3757260402
@Omega359 it looks good to me. Nice work @waynexia !
There's a lot of data now with the 4 instance types, so I fired the before and after results into claude to see if we can get an overview of what's going on
I didn't see any null results in the c6a.xlarge this time, did we already make some improvements to run on a low memory system?
```
================================================================================
DATAFUSION PERFORMANCE DELTA: v47.0 → v52.0
================================================================================
Format: Cold Δ% | Hot Δ%
Negative % = faster (improvement)
Positive % = slower (regression)
================================================================================
DATAFUSION SINGLE (Parquet, single)
================================================================================
Query c6a.2xlarge c6a.4xlarge c6a.xlarge c8g.4xlarge
-------------------------------------------------------------------------------------------------------------------------
Q0 -47.9% | -97.8% -36.7% | -98.3% -37.4% | -97.9% -37.5% | -97.7%
Q1 -16.3% | -53.1% -18.1% | -57.9% -24.1% | -45.4% -26.5% | -62.9%
Q2 -17.0% | -36.3% -16.1% | -48.0% -17.7% | -29.9% -10.2% | -40.1%
Q3 +377.2% | -31.2% -3.9% | -40.5% +197.0% | -17.3% +4.7% | -38.6%
Q4 +76.6% | -20.5% +1.1% | -13.8% +11.3% | -20.8% +7.8% | -26.1%
Q5 +95.0% | +1.7% +2.9% | +1.3% +20.5% | +6.7% -1.0% | -2.3%
Q6 -50.9% | -98.2% -40.4% | -98.5% -65.3% | -98.6% -52.9% | -98.0%
Q7 -30.1% | -55.4% -22.5% | -58.2% -39.2% | -55.6% -39.1% | -65.5%
Q8 +49.8% | -14.0% -4.3% | -4.8% -7.4% | -24.6% -6.8% | -8.4%
Q9 +74.0% | -8.0% +3.6% | -7.3% -4.2% | -24.1% +5.5% | +4.7%
Q10 +154.2% | -36.9% -10.9% | -29.5% +75.8% | -31.6% -7.8% | -31.3%
Q11 +184.5% | -38.7% +9.3% | -30.3% +61.9% | -43.8% +14.2% | -28.7%
Q12 +109.0% | -11.7% +7.1% | -2.2% +10.6% | -17.4% +13.7% | -11.5%
Q13 +95.0% | -14.3% -10.7% | -10.7% +49.4% | -21.5% -6.8% | -22.0%
Q14 +109.5% | -10.9% -2.3% | -2.4% +16.7% | -17.5% -6.1% | -11.8%
Q15 +13.9% | -27.5% -9.0% | -17.1% -29.3% | -37.8% -10.1% | -22.3%
Q16 +87.9% | -15.0% +0.2% | -5.8% +14.4% | -24.6% -3.0% | -19.2%
Q17 +97.4% | -1.6% -1.3% | -3.4% +38.6% | -3.3% -1.7% | -12.8%
Q18 +78.6% | -15.4% +0.2% | -3.6% +160.6% | N/A +0.3% | -12.2%
Q19 +344.0% | -31.4% -11.0% | -40.9% +146.2% | -18.2% -21.5% | -36.2%
Q20 +109.5% | +5.7% +1.6% | -8.2% +114.8% | +10.4% +1.2% | -4.9%
Q21 +107.0% | +0.9% -0.4% | -7.4% +107.7% | -7.0% -0.4% | -8.9%
Q22 +103.2% | +57.7% +0.4% | +15.4% +103.4% | +126.1% -0.0% | -10.7%
Q23 +100.5% | +117.1% -0.0% | -3.7% +101.7% | +111.0% -0.0% | -10.1%
Q24 +129.1% | -4.7% -0.6% | -9.6% +132.8% | -7.7% +0.0% | -22.2%
Q25 +193.5% | -9.3% -4.8% | -10.7% +66.5% | -12.9% +17.4% | -32.1%
Q26 +127.9% | -11.6% +0.1% | -18.4% +130.6% | -18.3% +9.3% | -27.3%
Q27 +111.3% | -15.1% +0.9% | -25.7% +111.0% | -27.3% +2.0% | -16.7%
Q28 -2.1% | -12.7% -4.1% | -6.8% -23.2% | -24.3% +3.6% | -5.8%
Q29 -7.8% | -2.6% -6.9% | -10.7% -4.4% | -4.2% -7.1% | -9.4%
Q30 +167.0% | -2.8% +21.4% | -2.4% +110.8% | -18.6% +22.7% | -10.0%
Q31 +131.3% | -0.5% +7.7% | -7.0% +127.5% | -14.1% +8.1% | -5.6%
Q32 +114.2% | -10.0% +6.0% | -2.7% +91.2% | N/A +6.9% | -6.1%
Q33 +93.3% | -12.1% -0.4% | -9.5% N/A | N/A -0.9% | -8.2%
Q34 +93.3% | -13.2% -0.2% | -10.1% N/A | N/A +0.3% | -8.2%
Q35 -14.7% | -16.1% -4.3% | -9.0% -24.6% | -29.8% -5.6% | -5.1%
Q36 -14.0% | -18.0% -12.7% | -31.8% -14.1% | -32.1% -19.3% | -26.0%
Q37 -13.2% | +12.2% -5.2% | +0.0% -8.5% | -6.1% -31.1% | -35.2%
Q38 -8.3% | -27.2% -14.6% | -37.0% -14.9% | -27.6% -9.6% | -29.4%
Q39 +9.2% | -10.2% -6.1% | -22.3% -11.0% | -18.0% -1.5% | -18.9%
Q40 -14.5% | -53.7% -22.2% | -49.2% -24.9% | -54.5% -23.3% | -49.6%
Q41 -23.9% | -52.0% -29.3% | -55.3% -27.6% | -59.8% -26.4% | -45.5%
Q42 -22.1% | -56.6% -26.6% | -55.4% -23.8% | -58.6% -29.2% | -54.1%
================================================================================
DATAFUSION PARTITIONED (Parquet, partitioned)
================================================================================
Query c6a.2xlarge c6a.4xlarge c6a.xlarge c8g.4xlarge
-------------------------------------------------------------------------------------------------------------------------
Q0 -23.5% | -90.5% -27.6% | -87.5% -38.7% | -94.2% -30.9% | -81.8%
Q1 -29.9% | -34.2% -29.3% | -34.7% -25.8% | -35.4% -37.1% | -17.5%
Q2 +162.4% | -21.6% -11.5% | -24.4% +69.7% | -24.3% -24.1% | -8.8%
Q3 +418.9% | -2.7% +43.0% | -9.8% +275.6% | -13.2% +90.7% | -16.2%
Q4 +77.7% | -16.9% +10.3% | -9.2% +6.8% | -24.0% +20.2% | -22.9%
Q5 +81.3% | +0.5% +8.4% | -3.8% +28.4% | -18.2% +7.9% | +8.6%
Q6 -57.7% | -94.7% -37.2% | -92.3% -67.7% | -96.9% -47.0% | -89.5%
Q7 -30.4% | -36.3% -20.0% | -33.8% -29.9% | -48.6% -37.4% | -18.2%
Q8 +73.2% | -14.3% -1.5% | -4.3% -2.5% | -25.3% +9.9% | +00.2%
Q9 +106.2% | -6.5% +27.0% | -2.8% +13.5% | -24.4% +29.1% | +15.2%
Q10 +169.5% | -29.2% +11.9% | -18.5% +85.7% | -39.5% +13.0% | -22.8%
Q11 +167.1% | -31.2% +25.4% | -14.3% +67.6% | -44.1% +28.0% | -23.8%
Q12 +90.9% | -14.4% +8.9% | -9.3% +15.2% | -34.1% +17.6% | -5.8%
Q13 +98.7% | -14.8% -4.0% | -7.3% +46.1% | -40.2% +0.4% | -3.0%
Q14 +99.1% | -13.9% +4.7% | -4.8% +26.9% | -31.0% +5.8% | -11.5%
Q15 +38.9% | -19.3% -4.3% | -12.6% -28.0% | -35.2% +5.8% | -19.8%
Q16 +93.1% | -16.5% +2.4% | -6.6% +16.4% | -31.9% +3.4% | -10.5%
Q17 +111.6% | -8.3% +2.7% | -3.5% +37.7% | -15.3% +3.9% | -3.2%
Q18 +85.9% | -13.7% +6.3% | -3.5% N/A | N/A +8.2% | -10.5%
Q19 +185.8% | -9.5% +5.8% | -12.3% +141.6% | -11.2% -3.5% | -8.4%
Q20 +116.6% | -6.0% +2.4% | -11.5% +111.5% | -12.0% +2.3% | +3.1%
Q21 +109.5% | -11.2% +1.2% | -10.6% +111.6% | -25.9% +0.9% | +3.5%
Q22 +106.1% | -17.6% +0.8% | -10.3% +106.9% | +111.8% +0.9% | -6.6%
Q23 +96.0% | +103.0% -4.8% | -17.8% +95.9% | +104.8% -2.9% | -17.2%
Q24 -47.9% | -73.9% -4.9% | -70.1% -54.4% | -84.3% -15.7% | -64.7%
Q25 +228.5% | -41.2% +50.5% | -37.2% +122.4% | -51.0% +65.5% | -30.8%
Q26 -45.7% | -78.5% -16.5% | -75.9% -54.4% | -85.3% -80.3% | -65.6%
Q27 +118.9% | -17.8% +6.8% | -12.7% +121.4% | -31.5% +8.1% | -9.6%
Q28 -0.5% | -14.6% -1.6% | -9.0% -23.0% | -24.7% -3.3% | -7.5%
Q29 -9.9% | -5.5% -5.4% | -1.5% -7.3% | -4.9% -9.1% | -8.4%
Q30 +192.3% | -16.0% +30.9% | -3.6% +95.0% | -19.5% +29.7% | +3.0%
Q31 +142.4% | -8.9% +12.6% | -1.9% +140.6% | -18.3% +13.0% | +3.1%
Q32 +126.3% | -7.9% +9.7% | -1.9% N/A | N/A +11.0% | -7.3%
Q33 +100.5% | -13.8% +0.1% | -4.9% N/A | N/A +0.7% | -5.3%
Q34 +101.4% | -13.9% +1.4% | -5.0% N/A | N/A +0.3% | -13.6%
Q35 -11.6% | -9.1% -6.3% | -6.8% -22.9% | -35.5% -11.3% | -2.6%
Q36 -51.7% | -54.3% -48.9% | -51.8% -48.9% | -57.5% -48.6% | -47.7%
Q37 -41.9% | -35.1% -36.0% | -32.7% -41.8% | -47.7% -36.1% | -19.5%
Q38 -45.4% | -47.3% -48.6% | -47.2% -48.0% | -59.4% -47.0% | -47.4%
Q39 -1.3% | -52.1% -50.3% | -50.6% +23.8% | -62.6% -46.8% | -43.8%
Q40 -39.3% | -54.2% -49.2% | -48.2% -9.3% | -62.0% -51.8% | -44.6%
Q41 -28.5% | -54.3% -46.2% | -56.1% -21.1% | -70.8% -49.3% | -50.0%
Q42 -46.0% | -57.6% -37.1% | -55.6% -38.4% | -70.0% -41.7% | -46.2%
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3760244312
Thank you, that makes sense to me.
Should we publish the xlarge case number with swap enabled (which is the current number in ClickHouse/ClickBench#749), or rerun with swap disabled? I still kept the instance and compiled artifact, so I can get one soon. I'm wondering how it was done previously. This time it would be oom-killed on those queries, and the benchmark script is interrupted. Is it done manually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3790400192
Looks great @waynexia 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3783933679
I've rerun on `xlarge` without swap and updated to that pr (in [this commit](https://github.com/ClickHouse/ClickBench/pull/749/changes/b296ec9288f3c3d91153e87502098a98ac78ba03)).
I got this result by simply commenting out those queries that would be oom-killed, to keep the bench script running. They are `q18`, `q32`, `q33` and `q34`. We can take a closer look at them in follow-up tasks like https://github.com/apache/datafusion/issues/19425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 52.0.0 [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on issue #19826:
URL: https://github.com/apache/datafusion/issues/19826#issuecomment-3784151875
I did a pair compare (xlarge, partitioned) using prebuilt binary from homebrew, the result looks good. Maybe we can use prebuilt in future runs.
<table>
<tr>
<td>
prebuilt
<td>
cargo build
<tr>
<td>
```
[0.053, 0.002, 0.002],
[0.153, 0.067, 0.067],
[0.904, 0.202, 0.199],
[2.576, 0.170, 0.170],
[3.141, 1.731, 1.738],
[3.311, 1.716, 1.739],
[0.057, 0.002, 0.002],
[0.165, 0.070, 0.069],
[3.444, 2.092, 2.085],
[4.618, 2.628, 2.451],
[2.417, 0.464, 0.466],
[2.600, 0.527, 0.530],
[3.308, 1.716, 1.455],
[6.430, 2.028, 2.057],
[3.452, 1.425, 1.409],
[2.843, 1.918, 1.909],
[6.819, 3.478, 3.432],
[6.771, 3.398, 3.646],
[null, null, null],
[1.924, 0.147, 0.145],
[20.899, 2.162, 2.184],
[23.984, 2.619, 2.616],
[45.659, 34.597, 31.576],
[110.961, 99.503, 94.832],
[1.247, 0.196, 0.189],
[3.304, 0.530, 0.529],
[1.251, 0.230, 0.213],
[21.169, 2.944, 3.024],
[31.841, 30.421, 30.234],
[1.604, 1.440, 1.438],
[7.554, 1.786, 1.767],
[15.036, 1.517, 1.549],
[null, null, null],
[null, null, null],
[null, null, null],
[3.100, 2.517, 2.537],
[0.161, 0.053, 0.053],
[0.115, 0.040, 0.036],
[0.162, 0.056, 0.052],
[0.590, 0.082, 0.077],
[0.154, 0.025, 0.024],
[0.142, 0.021, 0.021],
[0.094, 0.018, 0.019],
```
<td>
```
[0.050, 0.002, 0.002],
[0.155, 0.070, 0.069],
[0.916, 0.211, 0.210],
[2.559, 0.176, 0.177],
[3.135, 1.786, 1.855],
[3.332, 1.705, 1.709],
[0.053, 0.002, 0.002],
[0.165, 0.073, 0.073],
[3.476, 2.107, 2.106],
[4.591, 2.450, 2.461],
[2.405, 0.485, 0.461],
[2.598, 0.534, 0.576],
[3.340, 1.444, 1.455],
[6.839, 2.004, 2.061],
[3.427, 1.412, 1.403],
[2.831, 1.924, 1.911],
[6.857, 3.741, 3.456],
[6.659, 3.394, 3.398],
[null, null, null],
[1.927, 0.151, 0.150],
[20.884, 2.116, 2.132],
[23.982, 2.600, 2.573],
[45.662, 34.659, 32.172],
[111.062, 100.587, 94.511],
[1.224, 0.198, 0.193],
[3.302, 0.520, 0.519],
[1.246, 0.218, 0.213],
[21.202, 2.943, 2.926],
[30.980, 29.455, 29.697],
[1.574, 1.418, 1.408],
[7.496, 1.788, 1.786],
[15.047, 1.533, 1.524],
[null, null, null],
[null, null, null],
[null, null, null],
[2.984, 2.515, 2.492],
[0.169, 0.051, 0.063],
[0.116, 0.034, 0.034],
[0.170, 0.055, 0.054],
[0.591, 0.077, 0.102],
[0.132, 0.027, 0.022],
[0.144, 0.025, 0.025],
[0.097, 0.018, 0.018]
```
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org