You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ahmed Albalawi <ah...@capitalone.com.INVALID> on 2023/10/05 16:32:07 UTC
Spark Compatibility with Spring Boot 3.x
Hello team,
We are in the process of upgrading one of our apps to Spring Boot 3.x while
using Spark, and we have encountered an issue with Spark compatibility,
specifically with Jakarta Servlet. Spring Boot 3.x uses Jakarta Servlet
while Spark uses Javax Servlet. Can we get some guidance on how to upgrade
to Spring Boot 3.x while continuing to use Spark.
The specific error is listed below:
java.lang.NoClassDefFoundError: javax/servlet/Servlet
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
The error comes up when we try to run a mvn clean install, and the
issue is in our test cases. This issue happens specifically when we
build our spark session. The line of code it traces down to is as
follows:
*session = SparkSession.builder().sparkContext(SparkContext.getOrCreate(sparkConf)).getOrCreate();*
What we have tried:
- We noticed according to this post
<https://stackoverflow.com/questions/75690210/apache-spark-with-spring-boot-failed-to-start-exception-factory-method-javasp>,
there are no compatible versions of spark using version 5 of the
Jakarta Servlet API
- We've tried <https://stackoverflow.com/questions/76618374/spark-3-4-1-jakarta-servlet-6-0-0-compatibility-issue>
using the maven shade plugin to use jakarta instead of javax, but are
running into some other issues with this.
- We've also looked at the following
<https://stackoverflow.com/questions/75350944/dependecy-conflict-apache-spark-and-spring-boot>
to use jakarta 4.x with jersey 2.x and still have an issue with the
servlet
Please let us know if there are any solutions to this issue. Thanks!
--
*Ahmed Albalawi*
Senior Associate Software Engineer • EP2 Tech - CuRE
571-668-3911 • 1680 Capital One Dr.
______________________________________________________________________
The information contained in this e-mail may be confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Spark Compatibility with Spring Boot 3.x
Posted by Sean Owen <sr...@gmail.com>.
I think we already updated this in Spark 4. However for now you would have
to also include a JAR with the jakarta.* classes instead.
You are welcome to try Spark 4 now by building from master, but it's far
from release.
On Thu, Oct 5, 2023 at 11:53 AM Ahmed Albalawi
<ah...@capitalone.com.invalid> wrote:
> Hello team,
>
> We are in the process of upgrading one of our apps to Spring Boot 3.x
> while using Spark, and we have encountered an issue with Spark
> compatibility, specifically with Jakarta Servlet. Spring Boot 3.x uses
> Jakarta Servlet while Spark uses Javax Servlet. Can we get some guidance on
> how to upgrade to Spring Boot 3.x while continuing to use Spark.
>
> The specific error is listed below:
>
> java.lang.NoClassDefFoundError: javax/servlet/Servlet
> at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
> at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
>
> The error comes up when we try to run a mvn clean install, and the issue is in our test cases. This issue happens specifically when we build our spark session. The line of code it traces down to is as follows:
>
> *session = SparkSession.builder().sparkContext(SparkContext.getOrCreate(sparkConf)).getOrCreate();*
>
> What we have tried:
>
> - We noticed according to this post <https://stackoverflow.com/questions/75690210/apache-spark-with-spring-boot-failed-to-start-exception-factory-method-javasp>, there are no compatible versions of spark using version 5 of the Jakarta Servlet API
>
> - We've tried <https://stackoverflow.com/questions/76618374/spark-3-4-1-jakarta-servlet-6-0-0-compatibility-issue> using the maven shade plugin to use jakarta instead of javax, but are running into some other issues with this.
> - We've also looked at the following <https://stackoverflow.com/questions/75350944/dependecy-conflict-apache-spark-and-spring-boot> to use jakarta 4.x with jersey 2.x and still have an issue with the servlet
>
>
> Please let us know if there are any solutions to this issue. Thanks!
>
>
> --
> *Ahmed Albalawi*
>
> Senior Associate Software Engineer • EP2 Tech - CuRE
>
> 571-668-3911 • 1680 Capital One Dr.
> ------------------------------
>
> The information contained in this e-mail may be confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>
>
Re: Spark Compatibility with Spring Boot 3.x
Posted by Angshuman Bhattacharya <an...@capitalone.com.INVALID>.
Thanks Ahmed. I am trying to bring this up with Spark DE community
On Thu, Oct 5, 2023 at 12:32 PM Ahmed Albalawi <
ahmed.albalawi@capitalone.com> wrote:
> Hello team,
>
> We are in the process of upgrading one of our apps to Spring Boot 3.x
> while using Spark, and we have encountered an issue with Spark
> compatibility, specifically with Jakarta Servlet. Spring Boot 3.x uses
> Jakarta Servlet while Spark uses Javax Servlet. Can we get some guidance on
> how to upgrade to Spring Boot 3.x while continuing to use Spark.
>
> The specific error is listed below:
>
> java.lang.NoClassDefFoundError: javax/servlet/Servlet
> at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
> at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
>
> The error comes up when we try to run a mvn clean install, and the issue is in our test cases. This issue happens specifically when we build our spark session. The line of code it traces down to is as follows:
>
> *session = SparkSession.builder().sparkContext(SparkContext.getOrCreate(sparkConf)).getOrCreate();*
>
> What we have tried:
>
> - We noticed according to this post <https://urldefense.com/v3/__https://stackoverflow.com/questions/75690210/apache-spark-with-spring-boot-failed-to-start-exception-factory-method-javasp__;!!EFVe01R3CjU!fdH40mQbStFzaUfVJ7JR-ZDD8DFiYsPseqp-yRu1BAFs8Zc3PZGVzaqF39BbL5vGOwm3mXNcTF1_O_mk_uj9FaBlEplW-4Oh7jceadvuqg$>, there are no compatible versions of spark using version 5 of the Jakarta Servlet API
>
> - We've tried <https://urldefense.com/v3/__https://stackoverflow.com/questions/76618374/spark-3-4-1-jakarta-servlet-6-0-0-compatibility-issue__;!!EFVe01R3CjU!fdH40mQbStFzaUfVJ7JR-ZDD8DFiYsPseqp-yRu1BAFs8Zc3PZGVzaqF39BbL5vGOwm3mXNcTF1_O_mk_uj9FaBlEplW-4Oh7jeV6Df7lQ$> using the maven shade plugin to use jakarta instead of javax, but are running into some other issues with this.
> - We've also looked at the following <https://urldefense.com/v3/__https://stackoverflow.com/questions/75350944/dependecy-conflict-apache-spark-and-spring-boot__;!!EFVe01R3CjU!fdH40mQbStFzaUfVJ7JR-ZDD8DFiYsPseqp-yRu1BAFs8Zc3PZGVzaqF39BbL5vGOwm3mXNcTF1_O_mk_uj9FaBlEplW-4Oh7jeiYCkybw$> to use jakarta 4.x with jersey 2.x and still have an issue with the servlet
>
>
> Please let us know if there are any solutions to this issue. Thanks!
>
>
> --
> *Ahmed Albalawi*
>
> Senior Associate Software Engineer • EP2 Tech - CuRE
>
> 571-668-3911 • 1680 Capital One Dr.
>
______________________________________________________________________
The information contained in this e-mail may be confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.