You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "buraksenn (via GitHub)" <gi...@apache.org> on 2026/09/04 08:07:48 UTC
[PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
buraksenn opened a new pull request, #24945:
URL: https://github.com/apache/datafusion/pull/24945
## Which issue does this PR close?
- Closes #24621.
## Rationale for this change
Protobuf hooks that access fields individually can silently omit newly added
state. For JSON sinks, this caused an explicitly configured compression level
to revert to the default after a physical-plan protobuf roundtrip.
Exhaustive destructuring makes newly added source, sink, and wire fields compile
errors until their serialization behavior is explicitly considered.
## What changes are included in this PR?
- Exhaustively destructure `JsonSource` and `JsonSink` in their encoders.
- Exhaustively destructure their protobuf nodes in the decoders.
- Keep the active `JsonSink` hook exhaustive while centralizing field mapping in
its public `TryFrom<&JsonSink>` conversion.
- Add `compression_level` to the `JsonWriterOptions` protobuf message and
preserve it in both conversion directions.
- Strengthen the JSON sink roundtrip test to inspect non-default sink options,
file configuration, and sort order directly.
The protobuf change is additive and backward compatible.
## Are these changes tested?
Yes. The focused JSON source and sink roundtrip tests pass:
```bash
cargo test -p datafusion-proto --test proto_integration roundtrip_json
```
## Are there any user-facing changes?
Physical plans containing JSON sinks now preserve an explicitly configured
compression level across protobuf roundtrips. Older payloads without the new
field continue to decode with no explicit compression level.
There are no breaking API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5537884409
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `63.79310%` with `21 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.58%. Comparing base ([`9ef3d47`](https://app.codecov.io/gh/apache/datafusion/commit/9ef3d47225b6b3426a0a4e34518115a984a3ecc9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`59d0f47`](https://app.codecov.io/gh/apache/datafusion/commit/59d0f47a02b1ec18a0562db0cea8d3c0cc8a3c52?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 64 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/proto-common/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fproto-common%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1jb21tb24vc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-json/src/file\_format.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fdatasource-json%2Fsrc%2Ffile_format.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWpzb24vc3JjL2ZpbGVfZm9ybWF0LnJz) | 66.66% | [2 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-json/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&filepath=datafusion%2Fdatasource-json%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWpzb24vc3JjL3NvdXJjZS5ycw==) | 87.50% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24945?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24945 +/- ##
==========================================
+ Coverage 81.52% 81.58% +0.06%
==========================================
Files 1123 1123
Lines 405970 410945 +4975
Branches 405970 410945 +4975
==========================================
+ Hits 330983 335287 +4304
- Misses 55626 55912 +286
- Partials 19361 19746 +385
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24945?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5537697578
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-datasource-json v55.0.0 (current)
Built [ 43.940s] (current)
Parsing datafusion-datasource-json v55.0.0 (current)
Parsed [ 0.013s] (current)
Building datafusion-datasource-json v55.0.0 (baseline)
Built [ 42.757s] (baseline)
Parsing datafusion-datasource-json v55.0.0 (baseline)
Parsed [ 0.013s] (baseline)
Checking datafusion-datasource-json v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.103s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 88.175s] datafusion-datasource-json
Building datafusion-proto v55.0.0 (current)
Built [ 52.731s] (current)
Parsing datafusion-proto v55.0.0 (current)
Parsed [ 0.017s] (current)
Building datafusion-proto v55.0.0 (baseline)
Built [ 53.028s] (baseline)
Parsing datafusion-proto v55.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.134s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 107.293s] datafusion-proto
Building datafusion-proto-common v55.0.0 (current)
Built [ 21.613s] (current)
Parsing datafusion-proto-common v55.0.0 (current)
Parsed [ 0.045s] (current)
Building datafusion-proto-common v55.0.0 (baseline)
Built [ 21.685s] (baseline)
Parsing datafusion-proto-common v55.0.0 (baseline)
Parsed [ 0.046s] (baseline)
Checking datafusion-proto-common v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.339s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:587
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 45.720s] datafusion-proto-common
Building datafusion-proto-models v55.0.0 (current)
Built [ 24.549s] (current)
Parsing datafusion-proto-models v55.0.0 (current)
Parsed [ 0.124s] (current)
Building datafusion-proto-models v55.0.0 (baseline)
Built [ 24.381s] (baseline)
Parsing datafusion-proto-models v55.0.0 (baseline)
Parsed [ 0.125s] (baseline)
Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 2.011s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:587
field JsonWriterOptions.compression_level in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:587
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 52.899s] datafusion-proto-models
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24945:
URL: https://github.com/apache/datafusion/pull/24945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on code in PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#discussion_r3951150079
##########
datafusion/proto-common/src/generated/prost.rs:
##########
@@ -583,6 +583,8 @@ pub struct EmptyMessage {}
pub struct JsonWriterOptions {
#[prost(enumeration = "CompressionTypeVariant", tag = "1")]
pub compression: i32,
+ #[prost(uint32, optional, tag = "2")]
+ pub compression_level: ::core::option::Option<u32>,
Review Comment:
I've missed this thanks. Updated PR desc and added an upgrade note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#discussion_r3942963979
##########
datafusion/proto-common/src/generated/prost.rs:
##########
@@ -583,6 +583,8 @@ pub struct EmptyMessage {}
pub struct JsonWriterOptions {
#[prost(enumeration = "CompressionTypeVariant", tag = "1")]
pub compression: i32,
+ #[prost(uint32, optional, tag = "2")]
+ pub compression_level: ::core::option::Option<u32>,
Review Comment:
The optional protobuf field preserves binary compatibility, but adding this public Rust field breaks existing literals such as `JsonWriterOptions { compression: 0 }`. These now need `compression_level: None` or `..Default::default()`. Could we correct “There are no breaking API changes” in the PR description and add a short upgrade note?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): destructure JSON source and sink serde hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24945:
URL: https://github.com/apache/datafusion/pull/24945#issuecomment-5573065943
Thanks @buraksenn and @kumarUjjawal !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org