You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/05/13 00:19:54 UTC
[PR] Spark 3.4: Pass FileIO on Spark's read path [iceberg]
kevinjqliu opened a new pull request, #16307:
URL: https://github.com/apache/iceberg/pull/16307
Backport of #15683 (and length fix #16284) to `spark/v3.4`.
Introduces `SerializableFileIOWithSize` to broadcast a table's `FileIO` to executors alongside the table metadata. Provides a `KnownSizeEstimation` so Spark skips the expensive `SizeEstimator` walk during broadcast, and makes `close()` a no-op on executors so broadcast cleanup does not destroy the driver's FileIO.
### Adaptation note
v3.4's `BaseReader` still used the legacy `table.encryption().decrypt(...)` path. I switched that one method to `fileIO.bulkDecrypt(...)` to match v3.5/4.0/4.1, since the broadcast `FileIO` is now an `EncryptingFileIO` (combined in the constructor). All other files match the v3.5 patch byte-for-byte (with paths translated).
### Validation
- `./gradlew -DsparkVersions=3.4 :iceberg-spark:iceberg-spark-3.4_2.12:test --tests "*SerializableFileIOWithSize*"` (new test, passes)
- `./gradlew -DsparkVersions=3.4 :iceberg-spark:iceberg-spark-3.4_2.12:test --tests "org.apache.iceberg.spark.source.TestSparkReaderDeletes"` (passes)
- Compile-checked spark-extensions tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Pass FileIO on Spark's read path [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16307:
URL: https://github.com/apache/iceberg/pull/16307#issuecomment-4436514354
thanks for the review! since this is a backport and only for spark 3.4, im comfortable with merging it in as is 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Pass FileIO on Spark's read path [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #16307:
URL: https://github.com/apache/iceberg/pull/16307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org