You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "DebadityaHait (via GitHub)" <gi...@apache.org> on 2026/08/24 21:23:19 UTC
[PR] docs: document C2R cost for wide/nested schemas in tuning guide [datafusion-comet]
DebadityaHait opened a new pull request, #5458:
URL: https://github.com/apache/datafusion-comet/pull/5458
## Which issue does this PR close?
Closes #5455.
## Rationale for this change
The tuning guide documents the transitionRevert mechanism but never mentions schema shape as the cost driver. As reported in #5455, columnar-to-row conversion becomes very expensive for wide or deeply nested schemas, where reverting the entire stage can be cheaper than paying repeated transitions.
## What changes are included in this PR?
Extends the "Reducing Row/Columnar Conversion Overhead" section of the tuning guide with one short paragraph covering schema-shape-driven C2R cost and earlier-revert guidance using full config keys.
## How are these changes tested?
Docs-only change; validated with a clean local Sphinx build of the site (no warnings for the changed file). Config keys and defaults match their definitions in `CometConf.scala`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: document C2R cost for wide/nested schemas in tuning guide [datafusion-comet]
Posted by "DebadityaHait (via GitHub)" <gi...@apache.org>.
DebadityaHait commented on PR #5458:
URL: https://github.com/apache/datafusion-comet/pull/5458#issuecomment-5413521271
Thanks @comphead @sunchao — addressed in `d13c7135`: the guidance now sits under its own h3 heading ("Entire-Plan Fallback for Wide or Deeply Nested Schemas") and states explicitly that the fallback applies to the entire plan, with Comet removing its native operators rather than leaving a mix of native and fallback operators joined by repeated conversions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: document C2R cost for wide/nested schemas in tuning guide [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5458:
URL: https://github.com/apache/datafusion-comet/pull/5458#issuecomment-5414080030
Thanks, merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: document C2R cost for wide/nested schemas in tuning guide [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5458:
URL: https://github.com/apache/datafusion-comet/pull/5458#issuecomment-5412948324
@DebadityaHait could you address the comment from @comphead in the above? after that I'll merge this. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: document C2R cost for wide/nested schemas in tuning guide [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5458:
URL: https://github.com/apache/datafusion-comet/pull/5458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org