You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ArnabKarmakar1108 (via GitHub)" <gi...@apache.org> on 2026/08/04 06:57:18 UTC

[I] Spark: Remove duplicated TestTables and use core test helper [iceberg]

ArnabKarmakar1108 opened a new issue, #17508:
URL: https://github.com/apache/iceberg/issues/17508

   ### Feature Request / Improvement
   
   ## Problem
   
   Spark v3.5, v4.0, and v4.1 each have a duplicate `TestTables` helper (`spark/*/spark/src/test/java/org/apache/iceberg/spark/source/TestTables.java:42`) with a TODO to use the core version instead.
   
   Core already has a richer `org.apache.iceberg.TestTables` in `core/src/test/java/org/apache/iceberg/TestTables.java`. The Spark copies are outdated and risk drifting from core test behavior.
   
   Affected Spark tests: `TestBaseReader`, `TestSparkReadProjection`, `TestSparkMetadataColumns`, `TestSparkCatalog`.
   
   ## Proposed behavior
   
   1. Delete the three Spark `TestTables` copies.
   2. Switch affected tests to `org.apache.iceberg.TestTables` (already available via `iceberg-core` test artifacts).
   3. Update call sites for API differences:
      - `create(...)` requires a `formatVersion` argument
      - `load(name)` → `load(File temp, String name)`
      - `replaceMetadata(...)` is not in core — add it there or refactor `TestSparkReadProjection` to commit via `TableOperations`
   
   ## Scope
   
   - Test-only refactor across `spark/v3.5`, `spark/v4.0`, `spark/v4.1`
   - ~4 test classes per Spark version
   - May need small additions to core `TestTables` (e.g. `replaceMetadata`, public `readMetadata`)
   - `TestSparkCatalog` needs extra care: core keys tables by `(location, name)`, not name alone
   
   ## Notes
   
   - No production code changes or new dependencies.
   - Can be done one Spark version at a time or all three together.
   
   ### Query engine
   
   Spark
   
   ### Willingness to contribute
   
   - [x] I can contribute this improvement/feature independently
   - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark: Remove duplicated TestTables and use core test helper [iceberg]

Posted by "ArnabKarmakar1108 (via GitHub)" <gi...@apache.org>.
ArnabKarmakar1108 commented on issue #17508:
URL: https://github.com/apache/iceberg/issues/17508#issuecomment-5315126157

   Hi @charliec05, I notice you've already opened a PR for this issue. Since Im not(and not planning to anytime soon) working on this currently, you can go ahead and take up this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark: Remove duplicated TestTables and use core test helper [iceberg]

Posted by "charliec05 (via GitHub)" <gi...@apache.org>.
charliec05 commented on issue #17508:
URL: https://github.com/apache/iceberg/issues/17508#issuecomment-5312161822

   Hi, I would like to work on this. I plan to migrate Spark 4.1 first to the core TestTables helper, run the affected tests, and then apply the same changes to 4.0 and 3.5. Please let me know if this is already being worked on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org