You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2026/07/24 14:46:47 UTC

[PR] feat: Add MERGE INTO support to FFI TableProvider [datafusion]

wirybeaver opened a new pull request, #23867:
URL: https://github.com/apache/datafusion/pull/23867

   > [!NOTE]
   > This draft depends on #22988. Its branch is stacked on that PR, so GitHub will temporarily show the parent commits too. I will rebase it onto `main` after #22988 merges.
   
   ## Which issue does this PR close?
   
   Follow-up to #22988.
   
   ## Rationale for this change
   
   #22988 adds `TableProvider::merge_into`. Without matching FFI plumbing, a foreign `TableProvider` cannot expose an implementation of that method: the consumer side would use the trait's default unsupported behavior instead.
   
   ## What changes are included in this PR?
   
   - Add a `merge_into` function pointer to `FFI_TableProvider` and dispatch it from `ForeignTableProvider`.
   - Serialize and deserialize `MergeIntoOp` with the logical extension codec and pass the source as an `FFI_ExecutionPlan`.
   - Expose the proto helper needed to decode `MergeIntoOp` at the FFI boundary.
   - Add in-process forced-foreign coverage and a real cross-library ABI integration test.
   
   ## Are these changes tested?
   
   - `cargo fmt --all`
   - `cargo clippy --all-targets --all-features -- -D warnings`
   - `cargo test -p datafusion-ffi`
   - `cargo test -p datafusion-ffi --features integration-tests`
   - Required extended workspace test suite from `AGENTS.md` on the identical combined tree before the split
   
   ## Are there any user-facing changes?
   
   Yes. This adds a function-pointer field to the public `#[repr(C)] FFI_TableProvider` layout, so it is an ABI change and requires the `api change` label. FFI consumers must rebuild against the corresponding DataFusion major version.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add MERGE INTO support to FFI TableProvider [datafusion]

Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #23867:
URL: https://github.com/apache/datafusion/pull/23867#issuecomment-5071141352

   Maintainer note: this PR extends the public `#[repr(C)] FFI_TableProvider` layout with `merge_into`, so it needs the `api change` label. I attempted to add it, but my account does not have label permission.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add MERGE INTO support to FFI TableProvider [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23867:
URL: https://github.com/apache/datafusion/pull/23867#issuecomment-5071921363

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23867?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `84.13333%` with `119 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 80.72%. Comparing base ([`3f81613`](https://app.codecov.io/gh/apache/datafusion/commit/3f816137306112f75b412e2806abd24fed0b916f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f18ee8`](https://app.codecov.io/gh/apache/datafusion/commit/6f18ee8931ac02e3980e271db4d771a639a427fc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23867?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/sql/src/statement.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Fstatement.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3N0YXRlbWVudC5ycw==) | 87.72% | [16 Missing and 18 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/ffi/src/table\_provider.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fffi%2Fsrc%2Ftable_provider.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mZmkvc3JjL3RhYmxlX3Byb3ZpZGVyLnJz) | 79.16% | [10 Missing and 15 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/optimizer/src/analyzer/type\_coercion.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Ftype_coercion.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2FuYWx5emVyL3R5cGVfY29lcmNpb24ucnM=) | 81.90% | [10 Missing and 9 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 73.13% | [8 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...afusion/optimizer/src/analyzer/function\_rewrite.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Ffunction_rewrite.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2FuYWx5emVyL2Z1bmN0aW9uX3Jld3JpdGUucnM=) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/expr/src/logical\_plan/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/expr/src/logical\_plan/tree\_node.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Ftree_node.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vdHJlZV9ub2RlLnJz) | 83.33% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/proto/src/logical\_plan/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Flogical_plan%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvbG9naWNhbF9wbGFuL21vZC5ycw==) | 60.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/optimizer/src/rewrite\_set\_comparison.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Frewrite_set_comparison.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3Jld3JpdGVfc2V0X2NvbXBhcmlzb24ucnM=) | 88.88% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...timizer/src/simplify\_expressions/simplify\_exprs.rs](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fsimplify_expressions%2Fsimplify_exprs.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3NpbXBsaWZ5X2V4cHJlc3Npb25zL3NpbXBsaWZ5X2V4cHJzLnJz) | 90.90% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #23867      +/-   ##
   ==========================================
   + Coverage   80.71%   80.72%   +0.01%     
   ==========================================
     Files        1090     1090              
     Lines      370339   371123     +784     
     Branches   370339   371123     +784     
   ==========================================
   + Hits       298926   299596     +670     
   - Misses      53603    53648      +45     
   - Partials    17810    17879      +69     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23867?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add MERGE INTO support to FFI TableProvider [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23867:
URL: https://github.com/apache/datafusion/pull/23867#issuecomment-5071787742

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion v54.1.0 (current)
          Built [ 112.904s] (current)
        Parsing datafusion v54.1.0 (current)
         Parsed [   0.050s] (current)
       Building datafusion v54.1.0 (baseline)
          Built [ 107.631s] (baseline)
        Parsing datafusion v54.1.0 (baseline)
         Parsed [   0.037s] (baseline)
       Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.927s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 223.782s] datafusion
       Building datafusion-expr v54.1.0 (current)
          Built [  26.902s] (current)
        Parsing datafusion-expr v54.1.0 (current)
         Parsed [   0.080s] (current)
       Building datafusion-expr v54.1.0 (baseline)
          Built [  26.627s] (baseline)
        Parsing datafusion-expr v54.1.0 (baseline)
         Parsed [   0.081s] (baseline)
       Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   1.805s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  56.468s] datafusion-expr
       Building datafusion-ffi v54.1.0 (current)
          Built [  60.147s] (current)
        Parsing datafusion-ffi v54.1.0 (current)
         Parsed [   0.064s] (current)
       Building datafusion-ffi v54.1.0 (baseline)
          Built [  60.695s] (baseline)
        Parsing datafusion-ffi v54.1.0 (baseline)
         Parsed [   0.067s] (baseline)
       Checking datafusion-ffi v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.347s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip
   
   --- failure constructible_struct_adds_field: externally-constructible struct adds field ---
   
   Description:
   A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ForeignLibraryModule.create_merge_table in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:116
   
   --- warning repr_c_plain_struct_fields_reordered: struct fields reordered in repr(C) struct ---
   
   Description:
   A public repr(C) struct had its fields reordered. This can change the struct's memory layout, possibly breaking FFI use cases that depend on field position and order.
           ref: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/repr_c_plain_struct_fields_reordered.ron
   
   Failed in:
     FFI_TableProvider.statistics moved from position 6 to 7, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:154
     FFI_TableProvider.logical_codec moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:156
     FFI_TableProvider.version moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:166
     FFI_TableProvider.library_marker_id moved from position 12 to 13, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:175
     ForeignLibraryModule.create_physical_optimizer_rule moved from position 17 to 18, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:119
     ForeignLibraryModule.create_context_aware_optimizer_rule moved from position 18 to 19, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:121
     ForeignLibraryModule.version moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:123
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
        Warning produced 1 major and 0 minor level warnings
       Finished [ 123.070s] datafusion-ffi
       Building datafusion-optimizer v54.1.0 (current)
          Built [  26.599s] (current)
        Parsing datafusion-optimizer v54.1.0 (current)
         Parsed [   0.032s] (current)
       Building datafusion-optimizer v54.1.0 (baseline)
          Built [  26.282s] (baseline)
        Parsing datafusion-optimizer v54.1.0 (baseline)
         Parsed [   0.035s] (baseline)
       Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.227s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  54.018s] datafusion-optimizer
       Building datafusion-proto v54.1.0 (current)
          Built [  59.358s] (current)
        Parsing datafusion-proto v54.1.0 (current)
         Parsed [   0.019s] (current)
       Building datafusion-proto v54.1.0 (baseline)
          Built [  59.221s] (baseline)
        Parsing datafusion-proto v54.1.0 (baseline)
         Parsed [   0.021s] (baseline)
       Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.350s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 120.356s] datafusion-proto
       Building datafusion-session v54.1.0 (current)
          Built [  35.287s] (current)
        Parsing datafusion-session v54.1.0 (current)
         Parsed [   0.011s] (current)
       Building datafusion-session v54.1.0 (baseline)
          Built [  35.422s] (baseline)
        Parsing datafusion-session v54.1.0 (baseline)
         Parsed [   0.012s] (baseline)
       Checking datafusion-session v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.216s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  71.908s] datafusion-session
       Building datafusion-sql v54.1.0 (current)
          Built [  40.615s] (current)
        Parsing datafusion-sql v54.1.0 (current)
         Parsed [   0.033s] (current)
       Building datafusion-sql v54.1.0 (baseline)
          Built [  40.619s] (baseline)
        Parsing datafusion-sql v54.1.0 (baseline)
         Parsed [   0.036s] (baseline)
       Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.330s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  83.289s] datafusion-sql
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org