You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "PingLiuPing (via GitHub)" <gi...@apache.org> on 2025/11/04 14:08:52 UTC
[I] Move Iceberg partition transform from TableWriter node to a new Projection node [incubator-gluten]
PingLiuPing opened a new issue, #11023:
URL: https://github.com/apache/incubator-gluten/issues/11023
### Description
Currently, Iceberg partition transform is performed in the execution layer (within TableWrite operator). For better architecture and optimization opportunities, this computation should be moved to the optimizer/planner layer where it logically belongs. And this can be achieved by adding a projection node above TableWriter node and letting this node to perform the partition transform and send the pre-computed partition keys to TableWriter.
Doing so would allow for this computation to be optimized further by, for example, pushing it below the exchange and letting it run with higher concurrency than write itself. It would also provide more transparency for debugging as we would get runtime stats just for this piece of computation. It would allow users to see more clearly in the query plan what is happening allowing for more efficient debugging.
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Move Iceberg partition transform from TableWriter node to a new Projection node [incubator-gluten]
Posted by "mbasmanova (via GitHub)" <gi...@apache.org>.
mbasmanova commented on issue #11023:
URL: https://github.com/apache/incubator-gluten/issues/11023#issuecomment-3486234423
CC: @rui-mo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Move Iceberg partition transform from TableWriter node to a new Projection node [incubator-gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on issue #11023:
URL: https://github.com/apache/incubator-gluten/issues/11023#issuecomment-3486644423
Thanks for raising this issue. Will we need to do the same for Presto?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Move Iceberg partition transform from TableWriter node to a new Projection node [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #11023:
URL: https://github.com/apache/incubator-gluten/issues/11023#issuecomment-3488720619
Let's fix it until 15035 merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org