You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "buraksenn (via GitHub)" <gi...@apache.org> on 2026/08/08 05:42:26 UTC
[PR] refactor(proto): migrate CsvSource serde [datafusion]
buraksenn opened a new pull request, #24177:
URL: https://github.com/apache/datafusion/pull/24177
## Which issue does this PR close?
- Part of #23516.
## Rationale for this change
Part of epic #23494. Moves `CsvSource` protobuf serialization from the central dispatch into the source implementation.
## What changes are included in this PR?
Add protobuf serialization and deserialization to `CsvSource`, including the CSV format options represented by the existing wire format.
Repoint the live decode arm to `CsvSource::try_from_proto` and remove the old central encode arm. Keep `try_into_csv_scan_physical_plan` as a deprecated compatibility wrapper that delegates to the new implementation.
The protobuf wire format remains unchanged.
## Are these changes tested?
Yes. Added `roundtrip_csv_scan_preserves_format_options`, covering header, delimiter, quote, escape, comment, multiline values, and truncated rows.
## Are there any user-facing changes?
The existing `PhysicalPlanNodeExt` method remains available as a deprecated compatibility wrapper. There is no immediate API removal or wire-format change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate CsvSource serde [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24177:
URL: https://github.com/apache/datafusion/pull/24177#issuecomment-5224898115
Opened #24180 to track the scan options that are silently dropped on the wire (CSV `terminator` and compression type here, JSON `newline_delimited` in #24178) — pre-existing, not a blocker for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate CsvSource serde [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24177:
URL: https://github.com/apache/datafusion/pull/24177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate CsvSource serde [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24177:
URL: https://github.com/apache/datafusion/pull/24177#issuecomment-5224860224
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24177?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `68.60465%` with `27 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.05%. Comparing base ([`e0a05a6`](https://app.codecov.io/gh/apache/datafusion/commit/e0a05a693da6c56f97f001e0d5b3df1329b11ac2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`24d07fe`](https://app.codecov.io/gh/apache/datafusion/commit/24d07fec761a1e8405ea565f5391ed6938ab6829?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24177?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/datasource-csv/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/24177?src=pr&el=tree&filepath=datafusion%2Fdatasource-csv%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLWNzdi9zcmMvc291cmNlLnJz) | 74.35% | [6 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24177?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/physical\_plan/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/24177?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Fphysical_plan%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvcGh5c2ljYWxfcGxhbi9tb2QucnM=) | 12.50% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24177?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24177 +/- ##
==========================================
- Coverage 81.06% 81.05% -0.01%
==========================================
Files 1107 1106 -1
Lines 382191 382209 +18
Branches 382191 382209 +18
==========================================
- Hits 309807 309803 -4
- Misses 54082 54094 +12
- Partials 18302 18312 +10
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24177?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor(proto): migrate CsvSource serde [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24177:
URL: https://github.com/apache/datafusion/pull/24177#issuecomment-5224790706
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-datasource-csv v54.1.0 (current)
Built [ 48.762s] (current)
Parsing datafusion-datasource-csv v54.1.0 (current)
Parsed [ 0.012s] (current)
Building datafusion-datasource-csv v54.1.0 (baseline)
Built [ 43.946s] (baseline)
Parsing datafusion-datasource-csv v54.1.0 (baseline)
Parsed [ 0.012s] (baseline)
Checking datafusion-datasource-csv v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.106s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 94.221s] datafusion-datasource-csv
Building datafusion-proto v54.1.0 (current)
Built [ 61.018s] (current)
Parsing datafusion-proto v54.1.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-proto v54.1.0 (baseline)
Built [ 60.863s] (baseline)
Parsing datafusion-proto v54.1.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.235s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---
Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_marked_deprecated.ron
Failed in:
method try_into_csv_scan_physical_plan in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:1042
Summary semver requires new minor version: 0 major and 1 minor checks failed
Finished [ 123.598s] datafusion-proto
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org