You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/01/10 02:27:12 UTC
[D] Drop Spark 3.2 support [incubator-gluten]
GitHub user zhztheplayer created a discussion: Drop Spark 3.2 support
I've heard more and more people are upgrading their Spark clusters from 3.2 to higher versions. So perhaps it's necessary for us to discuss about dropping Spark 3.2 support from our main branch.
I have been auditing on the feasibility of this change because we can have chance to solve a long-existing issue https://github.com/apache/incubator-gluten/issues/540 after the migration. There might be other bonuses so feel free to supplement here.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] Drop Spark 3.2 support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
GitHub user FelixYBW added a comment to the discussion: Drop Spark 3.2 support
Our previous plan is to drop 3.2 support after we add Spark4.0 support.
We may need to share the info with all customers. Can you share the discussion in mail list? @weiting-chen
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491#discussioncomment-11801560
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] Drop Spark 3.2 support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
GitHub user FelixYBW added a comment to the discussion: Drop Spark 3.2 support
To clarify, after Gluten 1.4 release, we are going to disable Spark3.2's CI on github. The spark3.2 shimlayer will still be there.
The customer who uses spark3.2 may still run the 3.2 CI internally for every PR and fix the 3.2 bugs in upstream. I'd expect most merged PRs can still pass the 3.2 CI automatically.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491#discussioncomment-12131945
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] Drop Spark 3.2 support [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
GitHub user zhztheplayer edited a discussion: Drop Spark 3.2 support
I've heard more and more people are upgrading their Spark clusters from 3.2 to higher versions. So perhaps it's necessary for us to discuss about dropping Spark 3.2 support from our main branch.
I have been auditing on the feasibility of this change because we can have chance to solve a long-existing issue https://github.com/apache/incubator-gluten/issues/540 after the migration. The issue doesn't impact the stability of Gluten but could make the columnar batch management code complex to maintain, and could slightly impact performance. There might be other bonuses so feel free to supplement here.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] Drop Spark 3.2 support [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
GitHub user zhztheplayer edited a discussion: Drop Spark 3.2 support
I've heard more and more people are upgrading their Spark clusters from 3.2 to higher versions. So perhaps it's necessary for us to discuss about dropping Spark 3.2 support from our main branch.
I have been auditing on the feasibility of this change because we can have chance to solve a long-existing issue https://github.com/apache/incubator-gluten/issues/540 after the upgradation. The issue doesn't impact the stability of Gluten but could make the columnar batch management code complex to maintain, and could slightly impact performance. There might be other bonuses so feel free to supplement here.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] Drop Spark 3.2 support [incubator-gluten]
Posted by "weiting-chen (via GitHub)" <gi...@apache.org>.
GitHub user weiting-chen added a comment to the discussion: Drop Spark 3.2 support
Please check the email thread: https://lists.apache.org/thread/fgnv7v5sf9ol14lkssd9v47h6cvorcxc
The proposed plan will be
Gluten 1.4 with Spark 4.0 API support
Gluten 1.5 with Spark 4.0 UT support and Deprecate Spark3.2 support
For Spark 3.2 users, we will recommend to use Gluten 1.2 or 1.3 and a minor version upgrade with bug fix is on demand for better usage.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8491#discussioncomment-12115666
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org