You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2026/06/22 11:54:30 UTC
[I] improve: task serialization size is too large [auron]
wForget opened a new issue, #2351:
URL: https://github.com/apache/auron/issues/2351
**Is your feature request related to a problem? Please describe.**
https://github.com/wForget/benchmarks-spark-native/actions/runs/27946634258/job/82693031769
```
Running case: q64
......
WARN DAGScheduler: Broadcasting large task binary with size 140.0 MiB
```
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: task serialization size is too large [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 closed issue #2351: improve: task serialization size is too large
URL: https://github.com/apache/auron/issues/2351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: task serialization size is too large [auron]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2351:
URL: https://github.com/apache/auron/issues/2351#issuecomment-4768672213
The Broadcast object generated by `NativeBroadcastExchangeBase.doExecuteBroadcast` holds all rows data during serialization.
<img width="1637" height="819" alt="Image" src="https://github.com/user-attachments/assets/a12eef26-0ccd-46db-b077-c0b8ea99392f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: task serialization size is too large [auron]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2351:
URL: https://github.com/apache/auron/issues/2351#issuecomment-5519416725
@Sigma-Ma Thanks, this solution sounds good, please feel free to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: task serialization size is too large [auron]
Posted by "Sigma-Ma (via GitHub)" <gi...@apache.org>.
Sigma-Ma commented on issue #2351:
URL: https://github.com/apache/auron/issues/2351#issuecomment-5528606455
Hi @wForget, I’ve opened #2506 with the fix.
It replaces the value-capturing wrapper with a real Spark broadcast while keeping the native IPC path unchanged. I also added regression coverage for mixed native/Spark execution and serialized broadcast size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: task serialization size is too large [auron]
Posted by "Sigma-Ma (via GitHub)" <gi...@apache.org>.
Sigma-Ma commented on issue #2351:
URL: https://github.com/apache/auron/issues/2351#issuecomment-5495743841
Hi @wForget, I’d like to work on this issue.
The large task binary comes from the anonymous `Broadcast` in `doExecuteBroadcast`, which captures the transformed relation in the task closure.
I plan to use a real Spark broadcast for that relation while keeping the native IPC path unchanged, and add regression tests for serialization size and mixed native/Spark execution. Please let me know if this approach sounds good. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org