You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "buraksenn (via GitHub)" <gi...@apache.org> on 2026/08/08 18:52:17 UTC
[PR] refactor(proto): migrate AvroSource serde [datafusion]
buraksenn opened a new pull request, #24190:
URL: https://github.com/apache/datafusion/pull/24190
## Which issue does this PR close?
- Closes #23516.
## Rationale for this change
Part of epic #23494. Moves `AvroSource` protobuf serialization from the central dispatch into the source implementation.
## What changes are included in this PR?
Add protobuf serialization and deserialization to `AvroSource` and add the required `proto` feature wiring to `datafusion-datasource-avro`.
Repoint the feature-gated live decode arm to `AvroSource::try_from_proto` and remove the old central encode arm. Keep `try_into_avro_scan_physical_plan` as a deprecated compatibility wrapper that delegates to the new implementation.
The protobuf wire format remains unchanged.
## Are these changes tested?
Yes. Added `roundtrip_avro_scan`.
## Are there any user-facing changes?
The existing `PhysicalPlanNodeExt` method remains available as a deprecated compatibility wrapper. There is no immediate API removal or wire-format change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate AvroSource serde [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24190:
URL: https://github.com/apache/datafusion/pull/24190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate AvroSource serde [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24190:
URL: https://github.com/apache/datafusion/pull/24190#issuecomment-5227616897
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-datasource-avro v54.1.0 (current)
Built [ 44.875s] (current)
Parsing datafusion-datasource-avro v54.1.0 (current)
Parsed [ 0.012s] (current)
Building datafusion-datasource-avro v54.1.0 (baseline)
Built [ 40.730s] (baseline)
Parsing datafusion-datasource-avro v54.1.0 (baseline)
Parsed [ 0.011s] (baseline)
Checking datafusion-datasource-avro v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.094s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 87.139s] datafusion-datasource-avro
Building datafusion-proto v54.1.0 (current)
Built [ 61.319s] (current)
Parsing datafusion-proto v54.1.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-proto v54.1.0 (baseline)
Built [ 61.551s] (baseline)
Parsing datafusion-proto v54.1.0 (baseline)
Parsed [ 0.022s] (baseline)
Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.371s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---
Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_marked_deprecated.ron
Failed in:
method try_into_avro_scan_physical_plan in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:1043
Summary semver requires new minor version: 0 major and 1 minor checks failed
Finished [ 124.698s] datafusion-proto
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate AvroSource serde [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24190:
URL: https://github.com/apache/datafusion/pull/24190#issuecomment-5227808905
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24190?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `58.82353%` with `14 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.05%. Comparing base ([`eec8b94`](https://app.codecov.io/gh/apache/datafusion/commit/eec8b947c71a916385a28a3723753f4ffc7cd8d0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1b43c06`](https://app.codecov.io/gh/apache/datafusion/commit/1b43c061ddf91061eb58fbc8b002b7044074bf21?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24190?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/datasource-avro/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/24190?src=pr&el=tree&filepath=datafusion%2Fdatasource-avro%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWF2cm8vc3JjL3NvdXJjZS5ycw==) | 73.07% | [3 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/physical\_plan/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/24190?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Fphysical_plan%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvcGh5c2ljYWxfcGxhbi9tb2QucnM=) | 12.50% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24190 +/- ##
==========================================
- Coverage 81.05% 81.05% -0.01%
==========================================
Files 1106 1106
Lines 382287 382302 +15
Branches 382287 382302 +15
==========================================
+ Hits 309851 309862 +11
+ Misses 54121 54119 -2
- Partials 18315 18321 +6
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24190?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org