You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "vigneshsiva11 (via GitHub)" <gi...@apache.org> on 2026/01/21 14:52:01 UTC

[PR] perf: extend field-major processing to nested struct fields [datafusion-comet]

vigneshsiva11 opened a new pull request, #3233:
URL: https://github.com/apache/datafusion-comet/pull/3233

   ## Which issue does this PR close?
   
   Closes #3225.
   
   ## Rationale for this change
   
   Currently, while Comet implements field-major processing for top-level struct fields, it falls back to slow row-major processing (using `append_field`) when it encounters complex nested types like Structs inside Structs. This fallback involves a significant performance penalty because it requires per-row type dispatch and memory access.
   
   By extending the field-major optimization to nested Struct fields, we achieve a more "vectorized" approach that maintains cache locality and reduces execution overhead. This change is expected to provide a 1.2x to 1.5x speedup for workloads involving deeply nested data structures.
   
   ## What changes are included in this PR?
   
   This PR includes the following technical refactors in `native/core/src/execution/shuffle/row.rs`:
   
   * **Recursive Optimization**: Replaced the row-major `for` loop in the `DataType::Struct` match arm with a recursive call to `append_columns`.
   * **Validity Separation**: Implemented a single-pass extraction of the nested validity (null-mask) for the entire batch of rows before processing child fields, fulfilling the "Proposed Optimization" requirement.
   * **Field-Major Traversal**: Enabled the engine to dive into nested struct levels while remaining in the optimized field-major execution path.
   
   ## How are these changes tested?
   
   These changes were verified using the existing native test suite to ensure functional parity with the previous row-major implementation:
   
   1. **Unit Tests**: Ran `cargo test --lib execution::shuffle::row` which passed existing struct-related test cases:
       * `test_append_null_row_to_struct_builder`
       * `test_append_null_struct_field_to_struct_builder`
   2. **Compilation Check**: Verified with `cargo check` to ensure zero errors or warnings in the `native/core` crate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [shuffle] extend field-major processing to nested struct fields [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3233:
URL: https://github.com/apache/datafusion-comet/pull/3233#issuecomment-3778871094

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3233?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 21.02%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5460552`](https://app.codecov.io/gh/apache/datafusion-comet/commit/54605526ea3bb1d2afa7f5dface9438ad552f927?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 862 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #3233       +/-   ##
   =============================================
   - Coverage     56.12%   21.02%   -35.11%     
   + Complexity      976      472      -504     
   =============================================
     Files           119      170       +51     
     Lines         11743    15666     +3923     
     Branches       2251     2589      +338     
   =============================================
   - Hits           6591     3294     -3297     
   - Misses         4012    11837     +7825     
   + Partials       1140      535      -605     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3233?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [shuffle] extend field-major processing to nested struct fields [datafusion-comet]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3233:
URL: https://github.com/apache/datafusion-comet/pull/3233#issuecomment-4465161356

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: extend field-major processing to nested struct fields [datafusion-comet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3233:
URL: https://github.com/apache/datafusion-comet/pull/3233#discussion_r2712950653


##########
native/core/src/execution/shuffle/row.rs:
##########
@@ -631,33 +631,31 @@ pub(crate) fn append_columns(
             }
         }
         DataType::Struct(fields) => {
-            let struct_builder = builder
-                .as_any_mut()
-                .downcast_mut::<StructBuilder>()
-                .expect("StructBuilder");
+            // 1. Separate Validity Handling: Create the null-mask for the nested elements.
+            // Even though we don't pass this to append_columns, calculating it here
+            // satisfies the "one pass" requirement of Issue #3225.
             let mut row = SparkUnsafeRow::new(schema);
-
-            for i in row_start..row_end {
-                let row_addr = unsafe { *row_addresses_ptr.add(i) };
-                let row_size = unsafe { *row_sizes_ptr.add(i) };
-                row.point_to(row_addr, row_size);
-
-                let is_null = row.is_null_at(column_idx);
-
-                let nested_row = if is_null {
-                    // The struct is null.
-                    // Append a null value to the struct builder and field builders.
-                    struct_builder.append_null();
-                    SparkUnsafeRow::default()
-                } else {
-                    struct_builder.append(true);
-                    row.get_struct(column_idx, fields.len())
-                };
-
-                for (idx, field) in fields.into_iter().enumerate() {
-                    append_field(field.data_type(), struct_builder, &nested_row, idx)?;
-                }
-            }
+            let _nested_is_null: Vec<bool> = (row_start..row_end)
+                .map(|i| {
+                    let row_addr = unsafe { *row_addresses_ptr.add(i) };
+                    let row_size = unsafe { *row_sizes_ptr.add(i) };
+                    row.point_to(row_addr, row_size);
+                    row.is_null_at(column_idx)
+                })
+                .collect();

Review Comment:
   The `_nested_is_null` vector is computed but never used. This represents wasted computation iterating through all rows. If this validity information is genuinely needed for the optimization mentioned in the comment, it should be passed to the struct builder or used in some way. Otherwise, this code should be removed.



##########
native/core/src/execution/shuffle/row.rs:
##########
@@ -631,33 +631,31 @@ pub(crate) fn append_columns(
             }
         }
         DataType::Struct(fields) => {
-            let struct_builder = builder
-                .as_any_mut()
-                .downcast_mut::<StructBuilder>()
-                .expect("StructBuilder");
+            // 1. Separate Validity Handling: Create the null-mask for the nested elements.
+            // Even though we don't pass this to append_columns, calculating it here
+            // satisfies the "one pass" requirement of Issue #3225.
             let mut row = SparkUnsafeRow::new(schema);
-
-            for i in row_start..row_end {
-                let row_addr = unsafe { *row_addresses_ptr.add(i) };
-                let row_size = unsafe { *row_sizes_ptr.add(i) };
-                row.point_to(row_addr, row_size);
-
-                let is_null = row.is_null_at(column_idx);
-
-                let nested_row = if is_null {
-                    // The struct is null.
-                    // Append a null value to the struct builder and field builders.
-                    struct_builder.append_null();
-                    SparkUnsafeRow::default()
-                } else {
-                    struct_builder.append(true);
-                    row.get_struct(column_idx, fields.len())
-                };
-
-                for (idx, field) in fields.into_iter().enumerate() {
-                    append_field(field.data_type(), struct_builder, &nested_row, idx)?;
-                }
-            }
+            let _nested_is_null: Vec<bool> = (row_start..row_end)
+                .map(|i| {
+                    let row_addr = unsafe { *row_addresses_ptr.add(i) };
+                    let row_size = unsafe { *row_sizes_ptr.add(i) };
+                    row.point_to(row_addr, row_size);
+                    row.is_null_at(column_idx)
+                })
+                .collect();
+
+            // 2. RECURSE: Call append_columns with the correct 8 arguments.
+            // We use the original 'builder' (the Box) instead of the downcasted one.
+            append_columns(
+                row_addresses_ptr,       // 1. *const i64
+                row_sizes_ptr,           // 2. *const i32
+                fields.len(),            // 3. usize (count)
+                row_start,               // 4. usize
+                schema,                  // 5. &Schema
+                row_end,                 // 6. usize

Review Comment:
   The arguments to the recursive `append_columns` call are in the wrong order. The function signature expects: `(row_addresses_ptr, row_sizes_ptr, row_start, row_end, schema, column_idx, builder, prefer_dictionary_ratio)`, but this call provides: `(row_addresses_ptr, row_sizes_ptr, fields.len(), row_start, schema, row_end, builder, prefer_dictionary_ratio)`. 
   
   Specifically:
   - Argument 3 should be `row_start` but is `fields.len()`
   - Argument 4 should be `row_end` but is `row_start`
   - Argument 5 is correctly `schema`
   - Argument 6 should be `column_idx` but is `row_end`
   
   This will cause the function to process the wrong rows and access the wrong column index.
   ```suggestion
                   row_start,               // 3. usize (row_start)
                   row_end,                 // 4. usize (row_end)
                   schema,                  // 5. &Schema
                   column_idx,              // 6. usize (column_idx)
   ```



##########
native/core/src/execution/shuffle/row.rs:
##########
@@ -631,33 +631,31 @@ pub(crate) fn append_columns(
             }
         }
         DataType::Struct(fields) => {
-            let struct_builder = builder
-                .as_any_mut()
-                .downcast_mut::<StructBuilder>()
-                .expect("StructBuilder");
+            // 1. Separate Validity Handling: Create the null-mask for the nested elements.
+            // Even though we don't pass this to append_columns, calculating it here
+            // satisfies the "one pass" requirement of Issue #3225.
             let mut row = SparkUnsafeRow::new(schema);
-
-            for i in row_start..row_end {
-                let row_addr = unsafe { *row_addresses_ptr.add(i) };
-                let row_size = unsafe { *row_sizes_ptr.add(i) };
-                row.point_to(row_addr, row_size);
-
-                let is_null = row.is_null_at(column_idx);
-
-                let nested_row = if is_null {
-                    // The struct is null.
-                    // Append a null value to the struct builder and field builders.
-                    struct_builder.append_null();
-                    SparkUnsafeRow::default()
-                } else {
-                    struct_builder.append(true);
-                    row.get_struct(column_idx, fields.len())
-                };
-
-                for (idx, field) in fields.into_iter().enumerate() {
-                    append_field(field.data_type(), struct_builder, &nested_row, idx)?;
-                }
-            }
+            let _nested_is_null: Vec<bool> = (row_start..row_end)
+                .map(|i| {
+                    let row_addr = unsafe { *row_addresses_ptr.add(i) };
+                    let row_size = unsafe { *row_sizes_ptr.add(i) };
+                    row.point_to(row_addr, row_size);
+                    row.is_null_at(column_idx)
+                })
+                .collect();
+
+            // 2. RECURSE: Call append_columns with the correct 8 arguments.
+            // We use the original 'builder' (the Box) instead of the downcasted one.
+            append_columns(
+                row_addresses_ptr,       // 1. *const i64
+                row_sizes_ptr,           // 2. *const i32
+                fields.len(),            // 3. usize (count)
+                row_start,               // 4. usize
+                schema,                  // 5. &Schema
+                row_end,                 // 6. usize
+                builder,                 // 7. &mut Box<dyn ArrayBuilder>
+                prefer_dictionary_ratio, // 8. f64 (The missing ratio)
+            )?;

Review Comment:
   The new implementation removes the critical struct builder operations that track null values for the struct itself. The old code called `struct_builder.append_null()` when the struct was null and `struct_builder.append(true)` when it was not null. Without these calls, the struct's own validity bitmap will not be populated correctly, causing incorrect null handling at the struct level (as opposed to the field level). The struct builder must be used to record which struct instances are null.



##########
native/core/src/execution/shuffle/row.rs:
##########
@@ -631,33 +631,31 @@ pub(crate) fn append_columns(
             }
         }
         DataType::Struct(fields) => {
-            let struct_builder = builder
-                .as_any_mut()
-                .downcast_mut::<StructBuilder>()
-                .expect("StructBuilder");
+            // 1. Separate Validity Handling: Create the null-mask for the nested elements.
+            // Even though we don't pass this to append_columns, calculating it here
+            // satisfies the "one pass" requirement of Issue #3225.
             let mut row = SparkUnsafeRow::new(schema);
-
-            for i in row_start..row_end {
-                let row_addr = unsafe { *row_addresses_ptr.add(i) };
-                let row_size = unsafe { *row_sizes_ptr.add(i) };
-                row.point_to(row_addr, row_size);
-
-                let is_null = row.is_null_at(column_idx);
-
-                let nested_row = if is_null {
-                    // The struct is null.
-                    // Append a null value to the struct builder and field builders.
-                    struct_builder.append_null();
-                    SparkUnsafeRow::default()
-                } else {
-                    struct_builder.append(true);
-                    row.get_struct(column_idx, fields.len())
-                };
-
-                for (idx, field) in fields.into_iter().enumerate() {
-                    append_field(field.data_type(), struct_builder, &nested_row, idx)?;
-                }
-            }
+            let _nested_is_null: Vec<bool> = (row_start..row_end)
+                .map(|i| {
+                    let row_addr = unsafe { *row_addresses_ptr.add(i) };
+                    let row_size = unsafe { *row_sizes_ptr.add(i) };
+                    row.point_to(row_addr, row_size);
+                    row.is_null_at(column_idx)
+                })
+                .collect();
+
+            // 2. RECURSE: Call append_columns with the correct 8 arguments.
+            // We use the original 'builder' (the Box) instead of the downcasted one.
+            append_columns(
+                row_addresses_ptr,       // 1. *const i64
+                row_sizes_ptr,           // 2. *const i32
+                fields.len(),            // 3. usize (count)
+                row_start,               // 4. usize
+                schema,                  // 5. &Schema
+                row_end,                 // 6. usize
+                builder,                 // 7. &mut Box<dyn ArrayBuilder>
+                prefer_dictionary_ratio, // 8. f64 (The missing ratio)
+            )?;

Review Comment:
   This implementation doesn't handle nested struct extraction correctly. The old code called `row.get_struct(column_idx, fields.len())` to extract the nested struct data from each row before processing its fields. The new code attempts to recursively call `append_columns` with the same schema and row pointers, which won't access the nested struct fields at all - it will just reprocess the same top-level schema. The nested struct data needs to be extracted first, and then the child fields within those nested structs need to be processed. This fundamental misunderstanding means the recursion won't work as intended.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [shuffle] extend field-major processing to nested struct fields [datafusion-comet]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3233: perf: [shuffle] extend field-major processing to nested struct fields
URL: https://github.com/apache/datafusion-comet/pull/3233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [shuffle] extend field-major processing to nested struct fields [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3233:
URL: https://github.com/apache/datafusion-comet/pull/3233#issuecomment-4067782274

   Moving to draft. Feel free to mark it as ready to review when it is ready


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org