You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafusion.apache.org by Vignesh Siva <vi...@gmail.com> on 2026/01/22 12:01:40 UTC

[Discussion] Extending vectorized field-major processing for nested complex types (Issue #3225)

Hi everyone,I've been working on #3225 to extend field-major processing to
nested struct fields in Comet. My implementation focuses on separating
validity extraction from child field processing to improve cache locality
and reduce type-dispatch overhead.

As I finalize this PR, I have a question regarding the broader roadmap for
complex types:

Now that we have a recursive strategy for Struct fields, what are the
community's thoughts on applying a similar vectorized approach to List and
Map types? Specifically, what is the preferred pattern for handling
variable-length offsets in these collection types while staying within the
optimized field-major traversal path in the shuffle kernels?

I am a student contributor preparing for GSoc 2026 and would love to align
my current work with the long-term architectural goals for complex type
optimization in DataFusion Comet.

Best regards, Vignesh.

GitHub: vigneshsiva11