Prepare Spark 4.1 support

[Manu Zhang <ow...@gmail.com>]

Hi team,

We've been preparing for initial Spark 4.1 support[1] for a while, testing,
fixing bugs and compatibility issues. Meanwhile, the Spark community
is working towards 4.1.0 (RC1) release[2]. While there's a few remaining
issues to be resolved, Anton(aokolnychyi@apache.org) suggested getting it
into the main branch first such that it will be easier for the community to
collaborate on Spark 4.1 support.

If there's no concerns or objections, we are going to merge it.

1. https://github.com/apache/iceberg/pull/14155
<https://github.com/apache/iceberg/pull/14155>
2. https://lists.apache.org/thread/vcdz4fpmojbclogw1zntkotcnntjtjrt

Happy Thanksgiving!
Manu

Re: Prepare Spark 4.1 support

[Manu Zhang <ow...@gmail.com>]

Hi Anton,

Comet support for Spark 4.1 has just started[1]. I'm not sure about the
timeline but I'm devoted to adding 4.1 support in Comet as early as
possible.

For the current, I'm using
`org.apache.datafusion:comet-spark-spark4.0_2.13` as we used
`org.apache.datafusion:comet-spark-spark3.5_2.13`
for 4.0 before it's supported in Comet.

1.https://github.com/apache/datafusion-comet/pull/2829

Happy Thanksgiving!
Manu


On Thu, Nov 27, 2025 at 2:27 PM Anton Okolnychyi <ao...@gmail.com>
wrote:

> Thanks for driving this, Manu!
>
> What shall we do with Comet support? Is it broken for 4.1? If so, what's
> the timeline on the Comet side to add support for 4.1?
>
> - Anton
>
> вт, 25 лист. 2025 р. о 18:17 Manu Zhang <ow...@gmail.com> пише:
>
>> Hi team,
>>
>> We've been preparing for initial Spark 4.1 support[1] for a while,
>> testing, fixing bugs and compatibility issues. Meanwhile, the Spark
>> community is working towards 4.1.0 (RC1) release[2]. While there's a few
>> remaining issues to be resolved, Anton(aokolnychyi@apache.org) suggested
>> getting it into the main branch first such that it will be easier for the
>> community to collaborate on Spark 4.1 support.
>>
>> If there's no concerns or objections, we are going to merge it.
>>
>> 1. https://github.com/apache/iceberg/pull/14155
>> <https://github.com/apache/iceberg/pull/14155>
>> 2. https://lists.apache.org/thread/vcdz4fpmojbclogw1zntkotcnntjtjrt
>>
>> Happy Thanksgiving!
>> Manu
>>
>

Re: Prepare Spark 4.1 support

[Anton Okolnychyi <ao...@gmail.com>]

Thanks for driving this, Manu!

What shall we do with Comet support? Is it broken for 4.1? If so, what's
the timeline on the Comet side to add support for 4.1?

- Anton

вт, 25 лист. 2025 р. о 18:17 Manu Zhang <ow...@gmail.com> пише:

> Hi team,
>
> We've been preparing for initial Spark 4.1 support[1] for a while,
> testing, fixing bugs and compatibility issues. Meanwhile, the Spark
> community is working towards 4.1.0 (RC1) release[2]. While there's a few
> remaining issues to be resolved, Anton(aokolnychyi@apache.org) suggested
> getting it into the main branch first such that it will be easier for the
> community to collaborate on Spark 4.1 support.
>
> If there's no concerns or objections, we are going to merge it.
>
> 1. https://github.com/apache/iceberg/pull/14155
> <https://github.com/apache/iceberg/pull/14155>
> 2. https://lists.apache.org/thread/vcdz4fpmojbclogw1zntkotcnntjtjrt
>
> Happy Thanksgiving!
> Manu
>