You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huan233usc (via GitHub)" <gi...@apache.org> on 2026/09/04 21:30:02 UTC
[I] Iceberg Spark 4.2 feature tracking [iceberg]
huan233usc opened a new issue, #17959:
URL: https://github.com/apache/iceberg/issues/17959
### Feature Request / Improvement
Spark 4.2 adds lots of features and optimization which will be beneficial to Iceberg Spark Connector.
Creating this issue to track all of them. If you are interested in working on any of the items below, please open a dedicated sub-issue and link it back to this one.
(TODO: link Spark PR so anyone who is interested could get the context)
- Partition pruning @szehon-ho
- Transaction API
- Some pinning, refresh cache thing that Anton did
- Insert with Schema Evolution
- Auto CDC @huan233usc
- CREATE TABLE LIKE @rahulsmahadev
- SupportsSchemaEvolution capability filtering
Also some tests on existing behavior
- DSV2 cache invalidation/refresh
anything more
Thanks @szehon-ho and @rahulsmahadev for initial discussion and come up the list
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5546724483
Thanks @huan233usc for this, yes we had a lot of DSV2 work queued up. Everyone can also add more ideas as they come up (like any remaining item for ViewCatalog, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "rahulsmahadev (via GitHub)" <gi...@apache.org>.
rahulsmahadev commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5546939109
Yes, looking forward to 4.2 for a while. also need to add "Implement SupportsSchemaEvolution for automatic schema evolution- #17957
" into this list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "anuragmantri (via GitHub) via issues" <is...@iceberg.apache.org>.
anuragmantri commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6639400153
Had an offline chat with @szehon-ho. He said 4.2 is mostly ready. I will add it to 1:13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "Sir-Amirhossein (via GitHub)" <gi...@apache.org>.
Sir-Amirhossein commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6107451958
Hi, we're on Spark 4.2 and would like to use Iceberg 1.12 — but the released runtimes stop at 4.1 and there's no `iceberg-spark-runtime-4.2_2.13` artifact. So can we use `iceberg-spark-runtime-4.1_2.13` for spark 4.2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6161731850
> 14\. Hi, we're on Spark 4.2 and would like to use Iceberg 1.12 — but the released runtimes stop at 4.1 and there's no `iceberg-spark-runtime-4.2_2.13` artifact. So can we use `iceberg-spark-runtime-4.1_2.13` for spark 4.2 ?
Hi @Sir-Amirhossein I don't think 4.2 is supported in 1.12 -- The 4.2 support is still working in progress and validated by community
cc @nssalian @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5613684373
@huan233usc do you mind adding https://github.com/apache/iceberg/pull/16133 to the list? I'll add a fix soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5610564648
i add @aokolnychyi to also validate the iceberg spark 4.2 behavior for official release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "Sir-Amirhossein (via GitHub)" <gi...@apache.org>.
Sir-Amirhossein commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6121779365
> 14\. Hi, we're on Spark 4.2 and would like to use Iceberg 1.12 — but the released runtimes stop at 4.1 and there's no `iceberg-spark-runtime-4.2_2.13` artifact. So can we use `iceberg-spark-runtime-4.1_2.13` for spark 4.2 ?
I tested that and I realized it cannot be used because a class named View, I think, has changed in Spark 4.2.0 (Exception in thread "main" java.lang.IncompatibleClassChangeError: class org.apache.iceberg.spark.source.SparkView cannot implement org.apache.spark.sql.connector.catalog.View, because it is not an interface (org.apache.spark.sql.connector.catalog.View is in unnamed module of loader 'app')
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "dolfinus (via GitHub) via issues" <is...@iceberg.apache.org>.
dolfinus commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6573563425
Is it expected that Maven has `[iceberg-spark-runtime-4.2_2.13](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-4.2_2.13)` release [1.10.1.1-4.3.0-3](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-4.2_2.13/1.10.1.1-4.3.0-3)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5550517357
@huan233usc I see DSv2 table pinning and refresh semantics item [SPARK-58389](https://issues.apache.org/jira/browse/SPARK-58389), https://github.com/apache/spark/pull/57799, are targeted at Spark 4.3 — the new API isn't in the 4.2.0 artifact, so there's nothing to wire up in spark/v4.2 yet ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5876440272
> Also add tests for existing behavior:
> - DSv2 cache invalidation and refresh (spark#55416, spark#55536)
For https://github.com/apache/spark/pull/55536 (SPARK-54022 / SPARK-56617, DSv2 CACHE TABLE read semantics), it is already mirrored in TestCachedTableRefresh from the 4.1 -> 4.2 work. One scenario (a session ALTER TABLE invalidating the cache) was `@Disabled` pending SPARK-55631, which is fixed in 4.2.0, so #18298 re-enables it (now green under both the SparkCatalog and SparkSessionCatalog configs).
For https://github.com/apache/spark/pull/55416 (SPARK-54216, cache refresh with immutable Table instances), this PR was closed without merging and its JIRA is still open. The behavior looks already exercised by testCachedTableRefreshReflectsExternalChanges under the caching-catalog config, so I left it out; but I would be happy to add dedicated tests mirroring it if folks think that additional coverage is worth having.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5998293863
I would like to take the DSv2 Transaction API item (SPARK-55855). I opened a dedicated sub-issue to track it: #18384.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5610628881
CC: @qlong it would be great to track the variant related changes we want from Spark 4.2+ here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6164244223
@huan233usc, thanks for the note. The baseline support was added via some of the initial work for Spark 4.2 but the binaries weren't included in the Iceberg 1.12 since the community needs more time to make sure the changes for a new Spark version are clean. This issue tracks some of the pending items. If we close this out, I do think we can make a Spark 4.2 release in the upcoming Iceberg release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-5720518146
You’re right—thanks for catching this. I confirmed that the SPARK-58389 API is only available on master / Spark 4.3 and is not present in the 4.2.0 artifact, so there is nothing to wire up in `spark/v4.2`. I’ll remove this item from the Spark 4.2 tracker and track it for 4.3 instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Spark 4.2 feature tracking [iceberg]
Posted by "nssalian (via GitHub) via issues" <is...@iceberg.apache.org>.
nssalian commented on issue #17959:
URL: https://github.com/apache/iceberg/issues/17959#issuecomment-6637289631
@szehon-ho @anuragmantri could we target this for the 1.13 release. If yes, we could add it to the milestone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org