You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dmitri Bourlatchkov <di...@apache.org> on 2026/05/28 21:20:57 UTC

[DISCUSS] Code organization for Spark 3.x and 4.x

Hi All,

This is another discussion stemming from today's Community Sync call and PR
[4535].

Adding support for Spark 4 apparently produced a substantial amount of
"copied" code in [4535].

Points in favour of copy:

* Adjusting to differences between Spark versions is easier

* Dropping support for old Spark versions is easy (when they expire).

Points in favour of extracting common modules:

* Nice code organization. Common code is unit-tested once.

* Bug fixes in shared logic only need to be done in one place.

* Polaris does not appear to depend on deep Spark API (no query planning,
etc.) so differences between Spark versions can probably be handled by
allowing a small number of customization points in the common code.

I tend to prefer the second approach, that is factoring out common code and
sharing it between Spark 3.x and 4.x modules with the expectation that the
size of the common code is much larger than the size of the
version-specific code.

Thoughts?

[4535] https://github.com/apache/polaris/pull/4535

Thanks,
Dmitri.

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Yufei Gu <fl...@gmail.com>.
Thanks for raising the Iceberg convention, Ajantha! I think it's a good
idea to investigate it before extracting a common module for multiple Spark
versions.

Yufei


On Mon, Jun 1, 2026 at 6:07 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi all,
>
> I also prefer the approach to not have duplicated code.
>
> Looking at the `spark/src` and `integration/src` directories, I see 23
> byte-identical files, and 4 more files that have slight Spark-version
> specific differences that can be "deduplicated" with base classes plus
> version specific adapters.
>
> This appears to match Polaris's use of Spark, which is different from
> projects that deeply integrate with Spark or Flink planning and execution
> internals.
>
> Robert
>
>
> On Mon, Jun 1, 2026 at 7:48 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Dmitri,
> >
> > While I don't have a major concern with duplicating code in principle,
> > the main issue is the quantity of duplication. If the amount of
> > redundant code is large, it becomes significantly harder to maintain.
> >
> > For this reason, I prefer the second option of factoring out common code.
> >
> > Regards,
> > JB
> >
> > On Thu, May 28, 2026 at 11:21 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi All,
> > >
> > > This is another discussion stemming from today's Community Sync call
> and
> > PR
> > > [4535].
> > >
> > > Adding support for Spark 4 apparently produced a substantial amount of
> > > "copied" code in [4535].
> > >
> > > Points in favour of copy:
> > >
> > > * Adjusting to differences between Spark versions is easier
> > >
> > > * Dropping support for old Spark versions is easy (when they expire).
> > >
> > > Points in favour of extracting common modules:
> > >
> > > * Nice code organization. Common code is unit-tested once.
> > >
> > > * Bug fixes in shared logic only need to be done in one place.
> > >
> > > * Polaris does not appear to depend on deep Spark API (no query
> planning,
> > > etc.) so differences between Spark versions can probably be handled by
> > > allowing a small number of customization points in the common code.
> > >
> > > I tend to prefer the second approach, that is factoring out common code
> > and
> > > sharing it between Spark 3.x and 4.x modules with the expectation that
> > the
> > > size of the common code is much larger than the size of the
> > > version-specific code.
> > >
> > > Thoughts?
> > >
> > > [4535] https://github.com/apache/polaris/pull/4535
> > >
> > > Thanks,
> > > Dmitri.
> >
>

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Robert Stupp <sn...@snazy.de>.
Hi all,

I also prefer the approach to not have duplicated code.

Looking at the `spark/src` and `integration/src` directories, I see 23
byte-identical files, and 4 more files that have slight Spark-version
specific differences that can be "deduplicated" with base classes plus
version specific adapters.

This appears to match Polaris's use of Spark, which is different from
projects that deeply integrate with Spark or Flink planning and execution
internals.

Robert


On Mon, Jun 1, 2026 at 7:48 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi Dmitri,
>
> While I don't have a major concern with duplicating code in principle,
> the main issue is the quantity of duplication. If the amount of
> redundant code is large, it becomes significantly harder to maintain.
>
> For this reason, I prefer the second option of factoring out common code.
>
> Regards,
> JB
>
> On Thu, May 28, 2026 at 11:21 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi All,
> >
> > This is another discussion stemming from today's Community Sync call and
> PR
> > [4535].
> >
> > Adding support for Spark 4 apparently produced a substantial amount of
> > "copied" code in [4535].
> >
> > Points in favour of copy:
> >
> > * Adjusting to differences between Spark versions is easier
> >
> > * Dropping support for old Spark versions is easy (when they expire).
> >
> > Points in favour of extracting common modules:
> >
> > * Nice code organization. Common code is unit-tested once.
> >
> > * Bug fixes in shared logic only need to be done in one place.
> >
> > * Polaris does not appear to depend on deep Spark API (no query planning,
> > etc.) so differences between Spark versions can probably be handled by
> > allowing a small number of customization points in the common code.
> >
> > I tend to prefer the second approach, that is factoring out common code
> and
> > sharing it between Spark 3.x and 4.x modules with the expectation that
> the
> > size of the common code is much larger than the size of the
> > version-specific code.
> >
> > Thoughts?
> >
> > [4535] https://github.com/apache/polaris/pull/4535
> >
> > Thanks,
> > Dmitri.
>

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Dmitri,

While I don't have a major concern with duplicating code in principle,
the main issue is the quantity of duplication. If the amount of
redundant code is large, it becomes significantly harder to maintain.

For this reason, I prefer the second option of factoring out common code.

Regards,
JB

On Thu, May 28, 2026 at 11:21 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi All,
>
> This is another discussion stemming from today's Community Sync call and PR
> [4535].
>
> Adding support for Spark 4 apparently produced a substantial amount of
> "copied" code in [4535].
>
> Points in favour of copy:
>
> * Adjusting to differences between Spark versions is easier
>
> * Dropping support for old Spark versions is easy (when they expire).
>
> Points in favour of extracting common modules:
>
> * Nice code organization. Common code is unit-tested once.
>
> * Bug fixes in shared logic only need to be done in one place.
>
> * Polaris does not appear to depend on deep Spark API (no query planning,
> etc.) so differences between Spark versions can probably be handled by
> allowing a small number of customization points in the common code.
>
> I tend to prefer the second approach, that is factoring out common code and
> sharing it between Spark 3.x and 4.x modules with the expectation that the
> size of the common code is much larger than the size of the
> version-specific code.
>
> Thoughts?
>
> [4535] https://github.com/apache/polaris/pull/4535
>
> Thanks,
> Dmitri.

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Ajantha Bhat <aj...@gmail.com>.
We had a similar situation at Iceberg community long back and we decided to
keep duplicate code instead of keeping the common module.
https://github.com/apache/iceberg/pull/3313

Also, a similar approach has been taken for Flink modules too.

I cannot recall the decision points, I think it is because we can have
independent modules and faster development. More effort was needed for
maintaining common code.
Maybe someone who remembers the conversation from the Iceberg discussion
can share more on this.

- Ajantha








On Fri, May 29, 2026 at 9:00 AM Yong Zheng <yz...@apache.org> wrote:

> Hello Dmitri,
>
> Thanks for bring this one up. Personally, I like the second option as well
> as our code base around spark is really minimal. One thing I am not sure
> about is how do we decided which ones move to common module? Lets use
> couple examples:
> 1. For the ones with 100% identical and less likely to get change: for
> sure we can move those
> 2. For the ones with 100% identical for now but may change for the next
> version: how do we decide this? Do we move them back from common to spark
> version specific module? Or convert to adapter?
> 3. For the ones that already have difference, do we keep 2 files that are
> 80% identical and one per spark version or convert to adapter?
>
> Thanks,
> Yong
>
> On 2026/05/28 21:20:57 Dmitri Bourlatchkov wrote:
> > Hi All,
> >
> > This is another discussion stemming from today's Community Sync call and
> PR
> > [4535].
> >
> > Adding support for Spark 4 apparently produced a substantial amount of
> > "copied" code in [4535].
> >
> > Points in favour of copy:
> >
> > * Adjusting to differences between Spark versions is easier
> >
> > * Dropping support for old Spark versions is easy (when they expire).
> >
> > Points in favour of extracting common modules:
> >
> > * Nice code organization. Common code is unit-tested once.
> >
> > * Bug fixes in shared logic only need to be done in one place.
> >
> > * Polaris does not appear to depend on deep Spark API (no query planning,
> > etc.) so differences between Spark versions can probably be handled by
> > allowing a small number of customization points in the common code.
> >
> > I tend to prefer the second approach, that is factoring out common code
> and
> > sharing it between Spark 3.x and 4.x modules with the expectation that
> the
> > size of the common code is much larger than the size of the
> > version-specific code.
> >
> > Thoughts?
> >
> > [4535] https://github.com/apache/polaris/pull/4535
> >
> > Thanks,
> > Dmitri.
> >
>

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yong,

Thanks for the info!

DeltaHelper looks like a perfect example for the common/specific code
refactoring. It is a class owned by Polaris so we can define it as an
interface in the common code with different implementations specific to
Spark 3.x , 4.x etc.

I haven't looked too deeply into this code, but it seems doable... WDYT?

Cheers,
Dmitri.

On Sat, May 30, 2026 at 1:30 PM Yong Zheng <yz...@apache.org> wrote:

> Hello Dmitri,
>
> Yes. It may be not exact 20% diff but maybe we can use SparkCatalog.java
> as an example, there there are diff introduced by spark 4 API. The one that
> will actually have 20% diff will be DeltaHelper.java where there are some
> major changes. Detail can be found in
> https://github.com/apache/polaris/pull/4535#issuecomment-4551464567.
>
> Thanks,
> Yong Zheng
>
> On 2026/05/29 17:01:54 Dmitri Bourlatchkov wrote:
> > Hi Yong,
> >
> > Do you have an example of a file that is 80% identical between Spark 3
> and
> > 4? (sorry, I'm not very familiar with that codebase myself :)
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, May 28, 2026 at 11:30 PM Yong Zheng <yz...@apache.org> wrote:
> >
> > > Hello Dmitri,
> > >
> > > Thanks for bring this one up. Personally, I like the second option as
> well
> > > as our code base around spark is really minimal. One thing I am not
> sure
> > > about is how do we decided which ones move to common module? Lets use
> > > couple examples:
> > > 1. For the ones with 100% identical and less likely to get change: for
> > > sure we can move those
> > > 2. For the ones with 100% identical for now but may change for the next
> > > version: how do we decide this? Do we move them back from common to
> spark
> > > version specific module? Or convert to adapter?
> > > 3. For the ones that already have difference, do we keep 2 files that
> are
> > > 80% identical and one per spark version or convert to adapter?
> > >
> > > Thanks,
> > > Yong
> > >
> > > On 2026/05/28 21:20:57 Dmitri Bourlatchkov wrote:
> > > > Hi All,
> > > >
> > > > This is another discussion stemming from today's Community Sync call
> and
> > > PR
> > > > [4535].
> > > >
> > > > Adding support for Spark 4 apparently produced a substantial amount
> of
> > > > "copied" code in [4535].
> > > >
> > > > Points in favour of copy:
> > > >
> > > > * Adjusting to differences between Spark versions is easier
> > > >
> > > > * Dropping support for old Spark versions is easy (when they expire).
> > > >
> > > > Points in favour of extracting common modules:
> > > >
> > > > * Nice code organization. Common code is unit-tested once.
> > > >
> > > > * Bug fixes in shared logic only need to be done in one place.
> > > >
> > > > * Polaris does not appear to depend on deep Spark API (no query
> planning,
> > > > etc.) so differences between Spark versions can probably be handled
> by
> > > > allowing a small number of customization points in the common code.
> > > >
> > > > I tend to prefer the second approach, that is factoring out common
> code
> > > and
> > > > sharing it between Spark 3.x and 4.x modules with the expectation
> that
> > > the
> > > > size of the common code is much larger than the size of the
> > > > version-specific code.
> > > >
> > > > Thoughts?
> > > >
> > > > [4535] https://github.com/apache/polaris/pull/4535
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > >
> >
>

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Yong Zheng <yz...@apache.org>.
Hello Dmitri,

Yes. It may be not exact 20% diff but maybe we can use SparkCatalog.java as an example, there there are diff introduced by spark 4 API. The one that will actually have 20% diff will be DeltaHelper.java where there are some major changes. Detail can be found in https://github.com/apache/polaris/pull/4535#issuecomment-4551464567.

Thanks,
Yong Zheng

On 2026/05/29 17:01:54 Dmitri Bourlatchkov wrote:
> Hi Yong,
> 
> Do you have an example of a file that is 80% identical between Spark 3 and
> 4? (sorry, I'm not very familiar with that codebase myself :)
> 
> Thanks,
> Dmitri.
> 
> On Thu, May 28, 2026 at 11:30 PM Yong Zheng <yz...@apache.org> wrote:
> 
> > Hello Dmitri,
> >
> > Thanks for bring this one up. Personally, I like the second option as well
> > as our code base around spark is really minimal. One thing I am not sure
> > about is how do we decided which ones move to common module? Lets use
> > couple examples:
> > 1. For the ones with 100% identical and less likely to get change: for
> > sure we can move those
> > 2. For the ones with 100% identical for now but may change for the next
> > version: how do we decide this? Do we move them back from common to spark
> > version specific module? Or convert to adapter?
> > 3. For the ones that already have difference, do we keep 2 files that are
> > 80% identical and one per spark version or convert to adapter?
> >
> > Thanks,
> > Yong
> >
> > On 2026/05/28 21:20:57 Dmitri Bourlatchkov wrote:
> > > Hi All,
> > >
> > > This is another discussion stemming from today's Community Sync call and
> > PR
> > > [4535].
> > >
> > > Adding support for Spark 4 apparently produced a substantial amount of
> > > "copied" code in [4535].
> > >
> > > Points in favour of copy:
> > >
> > > * Adjusting to differences between Spark versions is easier
> > >
> > > * Dropping support for old Spark versions is easy (when they expire).
> > >
> > > Points in favour of extracting common modules:
> > >
> > > * Nice code organization. Common code is unit-tested once.
> > >
> > > * Bug fixes in shared logic only need to be done in one place.
> > >
> > > * Polaris does not appear to depend on deep Spark API (no query planning,
> > > etc.) so differences between Spark versions can probably be handled by
> > > allowing a small number of customization points in the common code.
> > >
> > > I tend to prefer the second approach, that is factoring out common code
> > and
> > > sharing it between Spark 3.x and 4.x modules with the expectation that
> > the
> > > size of the common code is much larger than the size of the
> > > version-specific code.
> > >
> > > Thoughts?
> > >
> > > [4535] https://github.com/apache/polaris/pull/4535
> > >
> > > Thanks,
> > > Dmitri.
> > >
> >
> 

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yong,

Do you have an example of a file that is 80% identical between Spark 3 and
4? (sorry, I'm not very familiar with that codebase myself :)

Thanks,
Dmitri.

On Thu, May 28, 2026 at 11:30 PM Yong Zheng <yz...@apache.org> wrote:

> Hello Dmitri,
>
> Thanks for bring this one up. Personally, I like the second option as well
> as our code base around spark is really minimal. One thing I am not sure
> about is how do we decided which ones move to common module? Lets use
> couple examples:
> 1. For the ones with 100% identical and less likely to get change: for
> sure we can move those
> 2. For the ones with 100% identical for now but may change for the next
> version: how do we decide this? Do we move them back from common to spark
> version specific module? Or convert to adapter?
> 3. For the ones that already have difference, do we keep 2 files that are
> 80% identical and one per spark version or convert to adapter?
>
> Thanks,
> Yong
>
> On 2026/05/28 21:20:57 Dmitri Bourlatchkov wrote:
> > Hi All,
> >
> > This is another discussion stemming from today's Community Sync call and
> PR
> > [4535].
> >
> > Adding support for Spark 4 apparently produced a substantial amount of
> > "copied" code in [4535].
> >
> > Points in favour of copy:
> >
> > * Adjusting to differences between Spark versions is easier
> >
> > * Dropping support for old Spark versions is easy (when they expire).
> >
> > Points in favour of extracting common modules:
> >
> > * Nice code organization. Common code is unit-tested once.
> >
> > * Bug fixes in shared logic only need to be done in one place.
> >
> > * Polaris does not appear to depend on deep Spark API (no query planning,
> > etc.) so differences between Spark versions can probably be handled by
> > allowing a small number of customization points in the common code.
> >
> > I tend to prefer the second approach, that is factoring out common code
> and
> > sharing it between Spark 3.x and 4.x modules with the expectation that
> the
> > size of the common code is much larger than the size of the
> > version-specific code.
> >
> > Thoughts?
> >
> > [4535] https://github.com/apache/polaris/pull/4535
> >
> > Thanks,
> > Dmitri.
> >
>

Re: [DISCUSS] Code organization for Spark 3.x and 4.x

Posted by Yong Zheng <yz...@apache.org>.
Hello Dmitri,

Thanks for bring this one up. Personally, I like the second option as well as our code base around spark is really minimal. One thing I am not sure about is how do we decided which ones move to common module? Lets use couple examples:
1. For the ones with 100% identical and less likely to get change: for sure we can move those
2. For the ones with 100% identical for now but may change for the next version: how do we decide this? Do we move them back from common to spark version specific module? Or convert to adapter?
3. For the ones that already have difference, do we keep 2 files that are 80% identical and one per spark version or convert to adapter?

Thanks,
Yong  

On 2026/05/28 21:20:57 Dmitri Bourlatchkov wrote:
> Hi All,
> 
> This is another discussion stemming from today's Community Sync call and PR
> [4535].
> 
> Adding support for Spark 4 apparently produced a substantial amount of
> "copied" code in [4535].
> 
> Points in favour of copy:
> 
> * Adjusting to differences between Spark versions is easier
> 
> * Dropping support for old Spark versions is easy (when they expire).
> 
> Points in favour of extracting common modules:
> 
> * Nice code organization. Common code is unit-tested once.
> 
> * Bug fixes in shared logic only need to be done in one place.
> 
> * Polaris does not appear to depend on deep Spark API (no query planning,
> etc.) so differences between Spark versions can probably be handled by
> allowing a small number of customization points in the common code.
> 
> I tend to prefer the second approach, that is factoring out common code and
> sharing it between Spark 3.x and 4.x modules with the expectation that the
> size of the common code is much larger than the size of the
> version-specific code.
> 
> Thoughts?
> 
> [4535] https://github.com/apache/polaris/pull/4535
> 
> Thanks,
> Dmitri.
>