You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "YannByron (via GitHub)" <gi...@apache.org> on 2024/08/12 08:00:36 UTC
[I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
YannByron opened a new issue, #3940:
URL: https://github.com/apache/paimon/issues/3940
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Support Spark4.0 (preview1)
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2348054052
@YannByron This approach just like Flink with two scala versions. I am OK with it~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2290841926
@ulysses-you we can discuss this here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2290889656
thank you @YannByron for the guide.
I looked at Spark 4.0.0-preview, the main challenge is scala2.13. Others like JDK17, inferface changes are not big issues.
For scala2.13, as far as I can see, Spark community paid a huge cost to support it and drop the scala2.12, and even for now there are some performance regression due to scala2.13, so I think it affects Paimon much.
For my self, I perfer to copy `paimon-spark-common` to a new module `paimon-spark-4.0`, so that we did not need to touch previous Spark version code. We can focus on the support with Spark 4.0.0 and higher version (may create `paimon-spark-4-common` if necessary).
cc @JingsongLi what do you think about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2345772482
The main issue of reuse module to me is we need to compile spark twice for different scala version. But I'm +1 for @YannByron if you are fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "awol2005ex (via GitHub)" <gi...@apache.org>.
awol2005ex commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2306127788
You can use "com.thoughtworks.enableIf" for multi versions of scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed issue #3940: [Feature] [Spark] Support Spark 4.0 (preview)
URL: https://github.com/apache/paimon/issues/3940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2342558439
Hi @ulysses-you @YannByron , I would like to ask whether `paimon-spark-4-common` and `paimon-spark-common` can reuse most of the code. I believe Spark 3 has very long-term support, and we also need to support Spark 4. If we end up copying a lot of code in this process, it will result in maintaining two separate codebases, which can be very costly. Therefore, my concern is whether we can reuse a significant portion of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] [Spark] Support Spark 4.0 (preview) [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on issue #3940:
URL: https://github.com/apache/paimon/issues/3940#issuecomment-2342749817
maybe we can allow `paimon-spark-common` to support both `scala.version` and `spark.version` properties(scala-2.12 and spark 3.5.2 by default), that make `paimon-spark-common` compatible spark `3.5` and `4.x`.Then provide a profile in top-level pom to compile `paimon-spark`.
This approach doesn't allow compile both spark 3.x and spark4.x at the same time and we have to modify something like CI. But this can avoid copying codes and make more reuse.
Meanwhile, `paimon-spark3-common` and `paimon-spark4-common` can be derived from `paimon-spark-common` easily if required.
@JingsongLi @ulysses-you WDYT~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org