You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "buraksenn (via GitHub)" <gi...@apache.org> on 2026/09/04 08:07:48 UTC

[PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

buraksenn opened a new pull request, #24945:
URL: https://github.com/apache/datafusion/pull/24945

   ## Which issue does this PR close?
   
   - Closes #24621.
   
   ## Rationale for this change
   
   Protobuf hooks that access fields individually can silently omit newly added
   state. For JSON sinks, this caused an explicitly configured compression level
   to revert to the default after a physical-plan protobuf roundtrip.
   
   Exhaustive destructuring makes newly added source, sink, and wire fields compile
   errors until their serialization behavior is explicitly considered.
   
   ## What changes are included in this PR?
   
   - Exhaustively destructure `JsonSource` and `JsonSink` in their encoders.
   - Exhaustively destructure their protobuf nodes in the decoders.
   - Keep the active `JsonSink` hook exhaustive while centralizing field mapping in
     its public `TryFrom<&JsonSink>` conversion.
   - Add `compression_level` to the `JsonWriterOptions` protobuf message and
     preserve it in both conversion directions.
   - Strengthen the JSON sink roundtrip test to inspect non-default sink options,
     file configuration, and sort order directly.
   
   The protobuf change is additive and backward compatible.
   
   ## Are these changes tested?
   
   Yes. The focused JSON source and sink roundtrip tests pass:
   
   ```bash
   cargo test -p datafusion-proto --test proto_integration roundtrip_json
   ```
   
   ## Are there any user-facing changes?
   
   Physical plans containing JSON sinks now preserve an explicitly configured
   compression level across protobuf roundtrips. Older payloads without the new
   field continue to decode with no explicit compression level.
   
   There are no breaking API changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5537884409

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `63.79310%` with `21 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 81.58%. Comparing base ([`9ef3d47`](https://app.codecov.io/gh/apache/datafusion/commit/9ef3d47225b6b3426a0a4e34518115a984a3ecc9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`59d0f47`](https://app.codecov.io/gh/apache/datafusion/commit/59d0f47a02b1ec18a0562db0cea8d3c0cc8a3c52?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 64 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/proto-common/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fproto-common%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1jb21tb24vc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/datasource-json/src/file\_format.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fdatasource-json%2Fsrc%2Ffile_format.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWpzb24vc3JjL2ZpbGVfZm9ybWF0LnJz) | 66.66% | [2 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/datasource-json/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fdatasource-json%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWpzb24vc3JjL3NvdXJjZS5ycw==) | 87.50% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #24945      +/-   ##
   ==========================================
   + Coverage   81.52%   81.58%   +0.06%     
   ==========================================
     Files        1123     1123              
     Lines      405970   410945    +4975     
     Branches   405970   410945    +4975     
   ==========================================
   + Hits       330983   335287    +4304     
   - Misses      55626    55912     +286     
   - Partials    19361    19746     +385     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5537697578

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-datasource-json v55.0.0 (current)
          Built [  43.940s] (current)
        Parsing datafusion-datasource-json v55.0.0 (current)
         Parsed [   0.013s] (current)
       Building datafusion-datasource-json v55.0.0 (baseline)
          Built [  42.757s] (baseline)
        Parsing datafusion-datasource-json v55.0.0 (baseline)
         Parsed [   0.013s] (baseline)
       Checking datafusion-datasource-json v55.0.0 -> v55.0.0 (no change; assume patch)
        Checked [   0.103s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [  88.175s] datafusion-datasource-json
       Building datafusion-proto v55.0.0 (current)
          Built [  52.731s] (current)
        Parsing datafusion-proto v55.0.0 (current)
         Parsed [   0.017s] (current)
       Building datafusion-proto v55.0.0 (baseline)
          Built [  53.028s] (baseline)
        Parsing datafusion-proto v55.0.0 (baseline)
         Parsed [   0.018s] (baseline)
       Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
        Checked [   0.134s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [ 107.293s] datafusion-proto
       Building datafusion-proto-common v55.0.0 (current)
          Built [  21.613s] (current)
        Parsing datafusion-proto-common v55.0.0 (current)
         Parsed [   0.045s] (current)
       Building datafusion-proto-common v55.0.0 (baseline)
          Built [  21.685s] (baseline)
        Parsing datafusion-proto-common v55.0.0 (baseline)
         Parsed [   0.046s] (baseline)
       Checking datafusion-proto-common v55.0.0 -> v55.0.0 (no change; assume patch)
        Checked [   1.339s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
     field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
     field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  45.720s] datafusion-proto-common
       Building datafusion-proto-models v55.0.0 (current)
          Built [  24.549s] (current)
        Parsing datafusion-proto-models v55.0.0 (current)
         Parsed [   0.124s] (current)
       Building datafusion-proto-models v55.0.0 (baseline)
          Built [  24.381s] (baseline)
        Parsing datafusion-proto-models v55.0.0 (baseline)
         Parsed [   0.125s] (baseline)
       Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
        Checked [   2.011s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:587
     field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:587
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  52.899s] datafusion-proto-models
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24945:
URL: https://github.com/apache/datafusion/pull/24945


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on code in PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#discussion_r3951150079


##########
datafusion/proto-common/src/generated/prost.rs:
##########
@@ -583,6 +583,8 @@ pub struct EmptyMessage {}
 pub struct JsonWriterOptions {
     #[prost(enumeration = "CompressionTypeVariant", tag = "1")]
     pub compression: i32,
+    #[prost(uint32, optional, tag = "2")]
+    pub compression_level: ::core::option::Option<u32>,

Review Comment:
   I've missed this thanks. Updated PR desc and added an upgrade note



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#discussion_r3942963979


##########
datafusion/proto-common/src/generated/prost.rs:
##########
@@ -583,6 +583,8 @@ pub struct EmptyMessage {}
 pub struct JsonWriterOptions {
     #[prost(enumeration = "CompressionTypeVariant", tag = "1")]
     pub compression: i32,
+    #[prost(uint32, optional, tag = "2")]
+    pub compression_level: ::core::option::Option<u32>,

Review Comment:
   The optional protobuf field preserves binary compatibility, but adding this public Rust field breaks existing literals such as `JsonWriterOptions { compression: 0 }`. These now need `compression_level: None` or `..Default::default()`. Could we correct “There are no breaking API changes” in the PR description and add a short upgrade note?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5573065943

   Thanks @buraksenn and @kumarUjjawal !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org