You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/07/20 11:46:46 UTC
[I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
alamb opened a new issue, #11567:
URL: https://github.com/apache/datafusion/issues/11567
### Is your feature request related to a problem or challenge?
Like https://github.com/apache/datafusion/issues/9404
DataFusion 40 has been released https://crates.io/crates/datafusion/40.0.0
It would be great to update ClickBench https://benchmark.clickhouse.com/ with runs from the latest version. It looks like we are still reporting numbers for DataFusion 36
### Describe the solution you'd like
Perhaps we can follow the model of https://github.com/ClickHouse/ClickBench/pull/178 (thanks @pmcgleenon ) or https://github.com/ClickHouse/ClickBench/pull/145 (thanks @kmitchener )
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2265469454
Thank you so much @pmcgleenon 🙏 -- I am pretty excited to complte our inprogress work (like stringview and high cardinality aggregates) and run these again with a newer version of DataFusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2258049992
I looked at some short queries and found one potential improvement https://github.com/apache/datafusion/issues/11719
I also looked at Q38
```sql
SELECT "URL", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate"::INT::DATE >= '2013-07-01' AND "EventDate"::INT::DATE <= '2013-07-31' AND "IsRefresh" = 0 AND "IsLink" <> 0 AND "IsDownload" = 0 GROUP BY "URL" ORDER BY PageViews DESC LIMIT 10 OFFSET 1000;
```
```shell
$ cargo run --release --bin dfbench -- clickbench --iterations 100 --path benchmarks/data/hits_partitioned --query 38
```
More than 50% of the time is spent doing snappy decoding (which we aren't likely to be able to improve)
<img width="1728" alt="Screenshot 2024-07-30 at 6 40 44 AM" src="https://github.com/user-attachments/assets/a1e53db1-6e67-4014-b4f5-77308a581c76">
12% of the time is reading string data from parquet (maybe stringview will help)
10% of the time is spent decoding parquet metadata
<img width="1728" alt="Screenshot 2024-07-30 at 6 44 17 AM" src="https://github.com/user-attachments/assets/c95aac55-0cac-4be7-a981-a3e3ce8c79ac">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2250224585
From what I understand, the previous [datafusion ClickBench runs](https://github.com/ClickHouse/ClickBench/blob/main/datafusion/README.md) have been on this AWS EC2 instance:
- c6a.4xlarge
- Amazon Linux 2 AMI
- Root 500GB gp2 SSD
- no EBS optimized
- no instance store
If we want to compare datafusion performance across different datafusion versions (and spot any improvements/degradations) then sticking to the same machine spec will allow us to do this.
If you check the [Clickbench results](https://benchmark.clickhouse.com/) many other databases also publish their results for the `c6a.4xlarge` AWS EC2 instance so we can compare datafusion results with DuckDB, ClickHouse, QuestDB and with many more.
IMO we should stick with the same AWS instance for future runs.
@alamb @xinlifoobar I'm happy to help out here if required!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2256663416
Thanks @alamb I'll create a PR on the Clickbench repo to update the results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2245279213
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2250246199
by the way this is what I found comparing AWS `c6a.4xlarge` and Azure `Standard_F16s_v2`. Looks like the CPU clock speed is different and there are some difference in the storage performance numbers
- https://cloudprice.net/aws/ec2/instances/c6a.4xlarge
- 16 vCPU
- CPU AMD 3.6 GHz
- 32 GB RAM
- https://cloudprice.net/vm/Standard_F16s_v2
- 16 vCPU
- CPU Intel 2.70GHz
- 32 GB RAM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2241100735
FWIW I think the benchmarks will improve dramatically once we have completed the StringView work that @XiangpengHao is leading https://github.com/apache/datafusion/issues/10918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2265439558
Hi @alamb
the [Clickbench PR](https://github.com/ClickHouse/ClickBench/pull/210) has been merged
the Datafusion version 40 results are now visible on the main ClickBench page https://benchmark.clickhouse.com/
<img width="1929" alt="Screenshot 2024-08-02 at 14 44 33" src="https://github.com/user-attachments/assets/4572c85f-d2fa-4db1-950f-56d68bfbe559">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2257031894
> It appears that the really low latency queries having gotten slower (perhaps due to increasing overhead in planning or runtime somewhere).
BTW I plan to spend some time tomorrow organizing / profiling the results to see if I can find some additional improvements to make.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2255581299
> Are these results inline with your expectations?
I would expect that we didn't see much performance difference between 40 and the other versions as we haven't done much on query performance recently.
It appears that the really low latency queries having gotten slower (perhaps due to increasing overhead in planning or runtime somewhere).
If these results are reproduceable, I do think we should publish them to clickbench
Thank you @pmcgleenon and @xinlifoobar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2250180970
I am not sure -- maybe @pmcgleenon could comment? I don't know if this is equivalent to the AWS machines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2257103794
That sounds great!
One benefit of the Clickbench results is that we can easily compare with other projects. Overall datafusion (partitioned) seems competitive with ClickHouse and DuckDB on similar hardware. In some cases the datafusion performance is better.
The results highlight a couple of scenarios where datafusion is behind (e.g. Q18, Q29, Q39 etc).
It will be exciting to see how the StringView work and any other improvements will change this picture in the future!
<img width="1010" alt="Screenshot 2024-07-29 at 23 08 02" src="https://github.com/user-attachments/assets/e947fe0b-9503-49b0-b879-d47d2dfa432d">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2254520675
@alamb @xinlifoobar Here are the results for df40 (attached is a file comparing 33, 34, 36 and 40)
Single
<img width="1282" alt="Screenshot 2024-07-28 at 14 26 24" src="https://github.com/user-attachments/assets/2e91e666-17f6-45da-a5aa-ca1fb325e780">
Partitioned
<img width="1473" alt="Screenshot 2024-07-28 at 14 26 09" src="https://github.com/user-attachments/assets/7ffbb98e-0418-4dd1-bc58-44ceb3ea7a2d">
[df40.zip](https://github.com/user-attachments/files/16403886/df40.zip)
Are these results inline with your expectations?
If so I can create a PR on Clickbench to update the datafusion results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2245316926
Sorry @alamb, I just found out that I could not create AWS account at this time. Is it fine to use Azure VM, e.g., F16sv2, instead? If not please unassign me...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2257970511
I expect StringView to help as well as @korowa 's https://github.com/apache/datafusion/pull/11627
In case anyone else is interested, I did an analysis of the various benchmark query properties here:
https://docs.google.com/spreadsheets/d/1NZuh_dEs9gX5uEp8AQ3DfvkNfC6bXFayQtFkjeKjNxQ/edit?gid=0#gid=0
(e.g. that is how I determine the relative cardinalities / what types of queries)
<img width="1522" alt="Screenshot 2024-07-30 at 5 55 37 AM" src="https://github.com/user-attachments/assets/0f22398c-ce8b-4da8-9769-ca962a100f09">
I think Q16/Q17/Q18 are all "high cardinality aggregates with mutli-column group keys"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #11567: Update ClickBench benchmarks with DataFusion 40
URL: https://github.com/apache/datafusion/issues/11567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2256764320
I've opened this PR for the Datafusion 40 results https://github.com/ClickHouse/ClickBench/pull/210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2258053998
I am pretty sure Q18 would be helped with https://github.com/apache/datafusion/issues/9403 -- maybe we'll find a way to do that shortly
```sql
SELECT "UserID", extract(minute FROM to_timestamp_seconds("EventTime")) AS m, "SearchPhrase", COUNT(*) FROM hits GROUP BY "UserID", m, "SearchPhrase" ORDER BY COUNT(*) DESC LIMIT 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Update ClickBench benchmarks with DataFusion 40 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11567:
URL: https://github.com/apache/datafusion/issues/11567#issuecomment-2250745174
> @alamb @xinlifoobar I'm happy to help out here if required!
That would be amazing 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org