You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/11/26 19:34:36 UTC
[I] [EPIC] Improve shuffle performance [datafusion-comet]
andygrove opened a new issue, #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123
I have been comparing Comet and Ballista performance for TPC-H q3. Both execution similar native plans. I am using the `comet-parquet-exec` branch which uses DataFusion's `ParquetExec`.
Most of the native execution time is spent in `ScanExec` which is reading Arrow batches from the JVM using Arrow FFI.
This epic is for improving shuffle / ScanExec performance.
Issues:
- https://github.com/apache/datafusion-comet/issues/1115
- https://github.com/apache/datafusion-comet/issues/977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2547006257
Here is a comparison of shuffle write metrics between Gluten and Comet. One thing I noticed is that Gluten is writing twice the amount of data when compared to Comet, so I wonder if there is a difference in compression or encoding that accounts for some of the time difference. I will keep investigating.
<img width="748" alt="Screenshot 2024-12-16 at 3 34 46 PM" src="https://github.com/user-attachments/assets/d9e3d7c5-9f69-4507-9ea5-49a818b813e7" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2536417956
It is interesting. For background task, do you see Gluten+Velox's shuffle is running as that (async?) and different to Spark/Comet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1123: [EPIC] Improve shuffle performance
URL: https://github.com/apache/datafusion-comet/issues/1123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2536254074
I ran some benchmarks this morning and found that with their respective shuffle managers disabled, Comet and Gluten(+Velox) have pretty similar performance (Gluten was 15% faster) but with shuffle managers enabled, Gluten was 176% faster. I am going to try and learn more about how Gluten+Velox implements shuffle as a background task. I think this validates that DataFusion and Velox likely have similar performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2504253986
> Do you mean when `ScanExec` is used as pseudo scan node to read shuffled data into next native execution?
Yes, exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2514934231
Here is an updated diagram showing that most of the native time is spent waiting for batches from the JVM and that the FFI overhead it not an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2549580106
Also here are updated metrics that now has the correct shuffle write time and also has the encoding time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2512310887
Update: The ScanExec time is from calling CometBatchIterator.next() so includes the time for it to fetch input batches as well as the FFI cost. It looks like the cost of fetching the batches is much more than the FFI cost. Perhaps this is really measuring the execution time of the input query, so could be misleading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2549431688
Spark by default uses lz4. shuffle codec is not configurable yet in Comet and it uses zstd. @andygrove Do you use same codec in the comparison?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2502413709
> Most of the native execution time in Comet is spent in ScanExec which is reading Arrow batches from the JVM using Arrow FFI.
Do you mean when `ScanExec` is used as pseudo scan node to read shuffled data into next native execution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2549578394
Thanks @Dandandan. I just discovered this morning that Gluten is using lz4. Comet is using zstd. I plan on trying lz4 in Comet as the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1123: Improve shuffle performance
URL: https://github.com/apache/datafusion-comet/issues/1123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2515239083
> Here is an updated diagram showing that most of the native time is spent waiting for batches from the JVM and that the FFI overhead it not an issue.
It makes more sense. I'm used to doubt that FFI overhead could be significant on performance number. It is designed to be lightweight to pass Arrow vectors around processes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2511747296
I created a Google document for collaborating on ideas around improving shuffle performance:
https://docs.google.com/document/d/1rx1ue7UZ4ljzic9Rc2kT-v35bfLB7Rhhe5FW1d0Sw4I/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2510299725
> If so, it makes some sense because Comet shuffle reader is still JVM based.
This is also an issue for shuffle writes. The child node of `ShuffleWriterExec` is always a `ScanNode` reading the output from a native plan.
We pay the FFI cost twice - once to import from native plan to JVM, then again to export to the shuffle native plan. We have the cost of schema serde in both directions. Perhaps there is a way to shortcut this and avoid a full serde because we do not need to read the batch in the JVM in this case, just pass it from one native plan to another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2549420485
One thing to try might be moving from zstd to lz4. The default zstd level (3 I believe) is quite slow for compression. Zstd can be tuned to be comparable in speed (fast mode), but then compression ratio will be lower as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2514984253
Breakdown of ScanExec by source:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Improve shuffle performance [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1123:
URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2960359323
I'm going to go ahead and close this now that almost all of the issues linked to this epic have been implemented. We have definitely seen a significant improvement in shuffle performance based on all of the work under this epic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org