You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/06/12 11:39:16 UTC
[PR] feat(spark): DataSource V2 connector, Scala (4/6) [datafusion-java]
timsaucer opened a new pull request, #107:
URL: https://github.com/apache/datafusion-java/pull/107
> **Stacked PR series (6 parts) — splitting the Spark DataSource V2 connector.**
> All six target `main`. They build on each other, so **review and merge in order** — until the earlier parts merge, this PR's diff includes their changes too.
>
> 1. Cargo workspace + `native-common` extraction
> 2. `datafusion-spark-bridge` Rust SDK
> 3. Spark connector Java SPI
> **➤ 4.** Spark DataSource V2 connector (Scala)
> 5. Bridge scaffold generator
> 6. End-to-end examples
## Purpose
The connector: a Spark **DataSource V2** implementation over the Java SPI and the bridge SDK. Covers DSv2 wiring (source/table/scan/builder), the per-partition columnar read path with Arrow→Spark conversion, V2 **predicate pushdown**, a **shared-scan mode** backed by a per-executor refcounted cache, and `SupportsReportPartitioning` for shuffle elision.
These share the scan-mode model and scan builder, so they land as one unit. Includes the connector test suite and module README.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): DataSource V2 connector, Scala (4/6) [datafusion-java]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed pull request #107: feat(spark): DataSource V2 connector, Scala (4/6)
URL: https://github.com/apache/datafusion-java/pull/107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): DataSource V2 connector, Scala (4/6) [datafusion-java]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #107:
URL: https://github.com/apache/datafusion-java/pull/107#issuecomment-4691250301
**Repo-layout docs follow-up (from #104 review):** #104 trimmed `docs/source/contributor-guide/development.md`. With the Scala connector landing here, the `spark/` bullet should describe the full `datafusion-java-spark` DataSource V2 connector (Scala + Java, pure JVM) rather than the Java-SPI-only stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): DataSource V2 connector, Scala (4/6) [datafusion-java]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #107:
URL: https://github.com/apache/datafusion-java/pull/107#issuecomment-4691139833
Heads-up from the foundation split (01-workspace-foundation): the RAT exclude `<exclude>**/META-INF/services/**</exclude>` was removed from the root `pom.xml` there because it was forward-referencing files that do not exist until this PR. This PR (04) is the first to add a `META-INF/services` registration file (`spark/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister`), which is a line-delimited FQCN list with no license header.
Action required: re-add the exclude to the root `pom.xml` RAT `<excludes>` block in this PR, or `apache-rat:check` will flag the file as unapproved once 01 is rebased in.
```xml
<!-- Spark DataSource SPI registration file (line-delimited FQCN list, no header) -->
<exclude>**/META-INF/services/**</exclude>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org