You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shrikant Prasad <sh...@gmail.com> on 2023/01/02 12:38:14 UTC
Spark migration from 2.3 to 3.0.1
Hi,
I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
The issue can be reproduced using below sample code:
object TestMain {
val session =
SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
def main(args: Array[String]): Unit = {
import session.implicits._
val a = *session.*sparkContext.parallelize(*Array*(("A",1),("B",2))).toDF("_
c1","_c2").*rdd*.map(x=> x(0).toString).collect()
*println*(a.mkString("|"))
}
}
It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
exception:
Caused by: org.apache.spark.SparkException: A master URL must be set in
your configuration
at
org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at TestMain$.<init>(TestMain.scala:7)
at TestMain$.<clinit>(TestMain.scala)
From the exception it appears that it tries to create spark session on
executor also in Spark 3 whereas its not created again on executor in Spark
2.3.
Can anyone help in identfying why there is this change in behavior?
Thanks and Regards,
Shrikant
--
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
I agree with you that it's not the recommended approach. But I just want to
understand which change caused this change in behavior. If you can point me
to some Jira in which this change was made, that would be greatly
appreciated.
Regards,
Shrikant
On Mon, 2 Jan 2023 at 9:46 PM, Sean Owen <sr...@gmail.com> wrote:
> Not true, you've never been able to use the SparkSession inside a Spark
> task. You aren't actually using it, if the application worked in Spark 2.x.
> Now, you need to avoid accidentally serializing it, which was the right
> thing to do even in Spark 2.x. Just move the sesion inside main(), not a
> member.
> Or what other explanation do you have? I don't understand.
>
> On Mon, Jan 2, 2023 at 10:10 AM Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> If that was the case and deserialized session would not work, the
>> application would not have worked.
>>
>> As per the logs and debug prints, in spark 2.3 the main object is not
>> getting deserialized in executor, otherise it would have failed then also.
>>
>> On Mon, 2 Jan 2023 at 9:15 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> It silently allowed the object to serialize, though the
>>> serialized/deserialized session would not work. Now it explicitly fails.
>>>
>>> On Mon, Jan 2, 2023 at 9:43 AM Shrikant Prasad <sh...@gmail.com>
>>> wrote:
>>>
>>>> Thats right. But the serialization would be happening in Spark 2.3
>>>> also, why we dont see this error there?
>>>>
>>>> On Mon, 2 Jan 2023 at 9:09 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> Oh, it's because you are defining "spark" within your driver object,
>>>>> and then it's getting serialized because you are trying to use TestMain
>>>>> methods in your program.
>>>>> This was never correct, but now it's an explicit error in Spark 3. The
>>>>> session should not be a member variable.
>>>>>
>>>>> On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Please see these logs. The error is thrown in executor:
>>>>>>
>>>>>> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0
>>>>>> (TID 0)
>>>>>>
>>>>>> java.lang.ExceptionInInitializerError
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>
>>>>>> at
>>>>>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>>>>>>
>>>>>> at
>>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>>>>>>
>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>
>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be set
>>>>>> in your configuration
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>>>>>>
>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>>>>>>
>>>>>> at TestMain$.<init>(TestMain.scala:12)
>>>>>>
>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>
>>>>>> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>>
>>>>>>> It's not running on the executor; that's not the issue. See your
>>>>>>> stack trace, where it clearly happens in the driver.
>>>>>>>
>>>>>>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <
>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>
>>>>>>>> Even if I set the master as yarn, it will not have access to rest
>>>>>>>> of the spark confs. It will need spark.yarn.app.id.
>>>>>>>>
>>>>>>>> The main issue is if its working as it is in Spark 2.3 why its not
>>>>>>>> working in Spark 3 i.e why the session is getting created on executor.
>>>>>>>> Another thing we tried is removing the df to rdd conversion just
>>>>>>>> for debug and it works in Spark 3.
>>>>>>>>
>>>>>>>> So, it might be something to do with df to rdd conversion or
>>>>>>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>>>>>>> But couldn't find the root cause.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shrikant
>>>>>>>>
>>>>>>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> So call .setMaster("yarn"), per the error
>>>>>>>>>
>>>>>>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <
>>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> We are running it in cluster deploy mode with yarn.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Shrikant
>>>>>>>>>>
>>>>>>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <
>>>>>>>>>> stevoo82@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>>>>>>
>>>>>>>>>>> The mater URL refers to as per java
>>>>>>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>>>>>>> according to where you want to run this
>>>>>>>>>>>
>>>>>>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <
>>>>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to migrate one spark application from Spark 2.3 to
>>>>>>>>>>>> 3.0.1.
>>>>>>>>>>>>
>>>>>>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>>>>>>
>>>>>>>>>>>> object TestMain {
>>>>>>>>>>>>
>>>>>>>>>>>> val session =
>>>>>>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>>>>>>
>>>>>>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>>>>>>
>>>>>>>>>>>> import session.implicits._
>>>>>>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>>>>>>> x(0).toString).collect()
>>>>>>>>>>>> *println*(a.mkString("|"))
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1
>>>>>>>>>>>> with below exception:
>>>>>>>>>>>>
>>>>>>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must
>>>>>>>>>>>> be set in your configuration
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>>>>>>
>>>>>>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>>>>>>
>>>>>>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>>>>>>
>>>>>>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From the exception it appears that it tries to create spark
>>>>>>>>>>>> session on executor also in Spark 3 whereas its not created again on
>>>>>>>>>>>> executor in Spark 2.3.
>>>>>>>>>>>>
>>>>>>>>>>>> Can anyone help in identfying why there is this change in
>>>>>>>>>>>> behavior?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Shrikant
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Shrikant Prasad
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shrikant Prasad
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shrikant Prasad
>>>>>>>>
>>>>>>> --
>>>>>> Regards,
>>>>>> Shrikant Prasad
>>>>>>
>>>>> --
>>>> Regards,
>>>> Shrikant Prasad
>>>>
>>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Sean Owen <sr...@gmail.com>.
Not true, you've never been able to use the SparkSession inside a Spark
task. You aren't actually using it, if the application worked in Spark 2.x.
Now, you need to avoid accidentally serializing it, which was the right
thing to do even in Spark 2.x. Just move the sesion inside main(), not a
member.
Or what other explanation do you have? I don't understand.
On Mon, Jan 2, 2023 at 10:10 AM Shrikant Prasad <sh...@gmail.com>
wrote:
> If that was the case and deserialized session would not work, the
> application would not have worked.
>
> As per the logs and debug prints, in spark 2.3 the main object is not
> getting deserialized in executor, otherise it would have failed then also.
>
> On Mon, 2 Jan 2023 at 9:15 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> It silently allowed the object to serialize, though the
>> serialized/deserialized session would not work. Now it explicitly fails.
>>
>> On Mon, Jan 2, 2023 at 9:43 AM Shrikant Prasad <sh...@gmail.com>
>> wrote:
>>
>>> Thats right. But the serialization would be happening in Spark 2.3 also,
>>> why we dont see this error there?
>>>
>>> On Mon, 2 Jan 2023 at 9:09 PM, Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> Oh, it's because you are defining "spark" within your driver object,
>>>> and then it's getting serialized because you are trying to use TestMain
>>>> methods in your program.
>>>> This was never correct, but now it's an explicit error in Spark 3. The
>>>> session should not be a member variable.
>>>>
>>>> On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Please see these logs. The error is thrown in executor:
>>>>>
>>>>> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0
>>>>> (TID 0)
>>>>>
>>>>> java.lang.ExceptionInInitializerError
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>
>>>>> at
>>>>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>
>>>>> at
>>>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>>>>>
>>>>> at
>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>>>>
>>>>> at
>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>>>>>
>>>>> at
>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>>>>>
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>>>>>
>>>>> at
>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>>>>>
>>>>> at
>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>>>>>
>>>>> at
>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>
>>>>> Caused by: org.apache.spark.SparkException: A master URL must be set
>>>>> in your configuration
>>>>>
>>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>>>>>
>>>>> at
>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>>>>>
>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>>>>>
>>>>> at TestMain$.<init>(TestMain.scala:12)
>>>>>
>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>
>>>>> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> It's not running on the executor; that's not the issue. See your
>>>>>> stack trace, where it clearly happens in the driver.
>>>>>>
>>>>>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <
>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>
>>>>>>> Even if I set the master as yarn, it will not have access to rest of
>>>>>>> the spark confs. It will need spark.yarn.app.id.
>>>>>>>
>>>>>>> The main issue is if its working as it is in Spark 2.3 why its not
>>>>>>> working in Spark 3 i.e why the session is getting created on executor.
>>>>>>> Another thing we tried is removing the df to rdd conversion just for
>>>>>>> debug and it works in Spark 3.
>>>>>>>
>>>>>>> So, it might be something to do with df to rdd conversion or
>>>>>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>>>>>> But couldn't find the root cause.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shrikant
>>>>>>>
>>>>>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>>>
>>>>>>>> So call .setMaster("yarn"), per the error
>>>>>>>>
>>>>>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <
>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> We are running it in cluster deploy mode with yarn.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Shrikant
>>>>>>>>>
>>>>>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <
>>>>>>>>> stevoo82@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>>>>>
>>>>>>>>>> The mater URL refers to as per java
>>>>>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>>>>>> according to where you want to run this
>>>>>>>>>>
>>>>>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <
>>>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to migrate one spark application from Spark 2.3 to
>>>>>>>>>>> 3.0.1.
>>>>>>>>>>>
>>>>>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>>>>>
>>>>>>>>>>> object TestMain {
>>>>>>>>>>>
>>>>>>>>>>> val session =
>>>>>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>>>>>
>>>>>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>>>>>
>>>>>>>>>>> import session.implicits._
>>>>>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>>>>>> x(0).toString).collect()
>>>>>>>>>>> *println*(a.mkString("|"))
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1
>>>>>>>>>>> with below exception:
>>>>>>>>>>>
>>>>>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be
>>>>>>>>>>> set in your configuration
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>>>>>
>>>>>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>>>>>
>>>>>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>>>>>
>>>>>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From the exception it appears that it tries to create spark
>>>>>>>>>>> session on executor also in Spark 3 whereas its not created again on
>>>>>>>>>>> executor in Spark 2.3.
>>>>>>>>>>>
>>>>>>>>>>> Can anyone help in identfying why there is this change in
>>>>>>>>>>> behavior?
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>
>>>>>>>>>>> Shrikant
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Shrikant Prasad
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Shrikant Prasad
>>>>>>>>>
>>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shrikant Prasad
>>>>>>>
>>>>>> --
>>>>> Regards,
>>>>> Shrikant Prasad
>>>>>
>>>> --
>>> Regards,
>>> Shrikant Prasad
>>>
>> --
> Regards,
> Shrikant Prasad
>
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
If that was the case and deserialized session would not work, the
application would not have worked.
As per the logs and debug prints, in spark 2.3 the main object is not
getting deserialized in executor, otherise it would have failed then also.
On Mon, 2 Jan 2023 at 9:15 PM, Sean Owen <sr...@gmail.com> wrote:
> It silently allowed the object to serialize, though the
> serialized/deserialized session would not work. Now it explicitly fails.
>
> On Mon, Jan 2, 2023 at 9:43 AM Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> Thats right. But the serialization would be happening in Spark 2.3 also,
>> why we dont see this error there?
>>
>> On Mon, 2 Jan 2023 at 9:09 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> Oh, it's because you are defining "spark" within your driver object, and
>>> then it's getting serialized because you are trying to use TestMain methods
>>> in your program.
>>> This was never correct, but now it's an explicit error in Spark 3. The
>>> session should not be a member variable.
>>>
>>> On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
>>> wrote:
>>>
>>>> Please see these logs. The error is thrown in executor:
>>>>
>>>> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0
>>>> (TID 0)
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>
>>>> at
>>>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>
>>>> at
>>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>>>>
>>>> at
>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>>>
>>>> at
>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>>>>
>>>> at
>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>>>>
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>>>>
>>>> at
>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>>>>
>>>> at
>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>>>>
>>>> at
>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>>>> your configuration
>>>>
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>>>>
>>>> at
>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>>>>
>>>> at
>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>>>>
>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>
>>>> at
>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>>>>
>>>> at TestMain$.<init>(TestMain.scala:12)
>>>>
>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>
>>>> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> It's not running on the executor; that's not the issue. See your stack
>>>>> trace, where it clearly happens in the driver.
>>>>>
>>>>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Even if I set the master as yarn, it will not have access to rest of
>>>>>> the spark confs. It will need spark.yarn.app.id.
>>>>>>
>>>>>> The main issue is if its working as it is in Spark 2.3 why its not
>>>>>> working in Spark 3 i.e why the session is getting created on executor.
>>>>>> Another thing we tried is removing the df to rdd conversion just for
>>>>>> debug and it works in Spark 3.
>>>>>>
>>>>>> So, it might be something to do with df to rdd conversion or
>>>>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>>>>> But couldn't find the root cause.
>>>>>>
>>>>>> Regards,
>>>>>> Shrikant
>>>>>>
>>>>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>>
>>>>>>> So call .setMaster("yarn"), per the error
>>>>>>>
>>>>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <
>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>
>>>>>>>> We are running it in cluster deploy mode with yarn.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shrikant
>>>>>>>>
>>>>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <
>>>>>>>> stevoo82@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>>>>
>>>>>>>>> The mater URL refers to as per java
>>>>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>>>>> according to where you want to run this
>>>>>>>>>
>>>>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <
>>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am trying to migrate one spark application from Spark 2.3 to
>>>>>>>>>> 3.0.1.
>>>>>>>>>>
>>>>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>>>>
>>>>>>>>>> object TestMain {
>>>>>>>>>>
>>>>>>>>>> val session =
>>>>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>>>>
>>>>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>>>>
>>>>>>>>>> import session.implicits._
>>>>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>>>>> x(0).toString).collect()
>>>>>>>>>> *println*(a.mkString("|"))
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>>>>>>> below exception:
>>>>>>>>>>
>>>>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be
>>>>>>>>>> set in your configuration
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>>>>
>>>>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>>>>
>>>>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>>>>
>>>>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From the exception it appears that it tries to create spark
>>>>>>>>>> session on executor also in Spark 3 whereas its not created again on
>>>>>>>>>> executor in Spark 2.3.
>>>>>>>>>>
>>>>>>>>>> Can anyone help in identfying why there is this change in
>>>>>>>>>> behavior?
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>
>>>>>>>>>> Shrikant
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shrikant Prasad
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shrikant Prasad
>>>>>>>>
>>>>>>> --
>>>>>> Regards,
>>>>>> Shrikant Prasad
>>>>>>
>>>>> --
>>>> Regards,
>>>> Shrikant Prasad
>>>>
>>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Sean Owen <sr...@gmail.com>.
It silently allowed the object to serialize, though the
serialized/deserialized session would not work. Now it explicitly fails.
On Mon, Jan 2, 2023 at 9:43 AM Shrikant Prasad <sh...@gmail.com>
wrote:
> Thats right. But the serialization would be happening in Spark 2.3 also,
> why we dont see this error there?
>
> On Mon, 2 Jan 2023 at 9:09 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> Oh, it's because you are defining "spark" within your driver object, and
>> then it's getting serialized because you are trying to use TestMain methods
>> in your program.
>> This was never correct, but now it's an explicit error in Spark 3. The
>> session should not be a member variable.
>>
>> On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
>> wrote:
>>
>>> Please see these logs. The error is thrown in executor:
>>>
>>> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0
>>> (TID 0)
>>>
>>> java.lang.ExceptionInInitializerError
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>
>>> at
>>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>
>>> at
>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>
>>> at
>>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>>>
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>>
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>>>
>>> at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>>
>>> at
>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>>>
>>> at
>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>>>
>>> at
>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>>>
>>> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>>>
>>> at
>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>>>
>>> at
>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>>>
>>> at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>>> your configuration
>>>
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>>>
>>> at
>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>>>
>>> at
>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>>>
>>> at scala.Option.getOrElse(Option.scala:189)
>>>
>>> at
>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>>>
>>> at TestMain$.<init>(TestMain.scala:12)
>>>
>>> at TestMain$.<clinit>(TestMain.scala)
>>>
>>> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> It's not running on the executor; that's not the issue. See your stack
>>>> trace, where it clearly happens in the driver.
>>>>
>>>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Even if I set the master as yarn, it will not have access to rest of
>>>>> the spark confs. It will need spark.yarn.app.id.
>>>>>
>>>>> The main issue is if its working as it is in Spark 2.3 why its not
>>>>> working in Spark 3 i.e why the session is getting created on executor.
>>>>> Another thing we tried is removing the df to rdd conversion just for
>>>>> debug and it works in Spark 3.
>>>>>
>>>>> So, it might be something to do with df to rdd conversion or
>>>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>>>> But couldn't find the root cause.
>>>>>
>>>>> Regards,
>>>>> Shrikant
>>>>>
>>>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> So call .setMaster("yarn"), per the error
>>>>>>
>>>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <
>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>
>>>>>>> We are running it in cluster deploy mode with yarn.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shrikant
>>>>>>>
>>>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>>>
>>>>>>>> The mater URL refers to as per java
>>>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>>>> according to where you want to run this
>>>>>>>>
>>>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <
>>>>>>>> shrikant.ism@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to migrate one spark application from Spark 2.3 to
>>>>>>>>> 3.0.1.
>>>>>>>>>
>>>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>>>
>>>>>>>>> object TestMain {
>>>>>>>>>
>>>>>>>>> val session =
>>>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>>>
>>>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>>>
>>>>>>>>> import session.implicits._
>>>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>>>> x(0).toString).collect()
>>>>>>>>> *println*(a.mkString("|"))
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>>>>>> below exception:
>>>>>>>>>
>>>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be
>>>>>>>>> set in your configuration
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>>>
>>>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>>>
>>>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>>>
>>>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From the exception it appears that it tries to create spark
>>>>>>>>> session on executor also in Spark 3 whereas its not created again on
>>>>>>>>> executor in Spark 2.3.
>>>>>>>>>
>>>>>>>>> Can anyone help in identfying why there is this change in
>>>>>>>>> behavior?
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>>
>>>>>>>>> Shrikant
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Shrikant Prasad
>>>>>>>>>
>>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shrikant Prasad
>>>>>>>
>>>>>> --
>>>>> Regards,
>>>>> Shrikant Prasad
>>>>>
>>>> --
>>> Regards,
>>> Shrikant Prasad
>>>
>> --
> Regards,
> Shrikant Prasad
>
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
Thats right. But the serialization would be happening in Spark 2.3 also,
why we dont see this error there?
On Mon, 2 Jan 2023 at 9:09 PM, Sean Owen <sr...@gmail.com> wrote:
> Oh, it's because you are defining "spark" within your driver object, and
> then it's getting serialized because you are trying to use TestMain methods
> in your program.
> This was never correct, but now it's an explicit error in Spark 3. The
> session should not be a member variable.
>
> On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> Please see these logs. The error is thrown in executor:
>>
>> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
>> 0)
>>
>> java.lang.ExceptionInInitializerError
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>>
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>>
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>
>> at
>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>>
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>>
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>
>> at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>
>> at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>>
>> at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>>
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>>
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>>
>> at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>>
>> at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>>
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>>
>> at
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>>
>> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>>
>> at
>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>>
>> at
>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>>
>> at
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>> your configuration
>>
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>>
>> at
>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>>
>> at
>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>>
>> at scala.Option.getOrElse(Option.scala:189)
>>
>> at
>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>>
>> at TestMain$.<init>(TestMain.scala:12)
>>
>> at TestMain$.<clinit>(TestMain.scala)
>>
>> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> It's not running on the executor; that's not the issue. See your stack
>>> trace, where it clearly happens in the driver.
>>>
>>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
>>> wrote:
>>>
>>>> Even if I set the master as yarn, it will not have access to rest of
>>>> the spark confs. It will need spark.yarn.app.id.
>>>>
>>>> The main issue is if its working as it is in Spark 2.3 why its not
>>>> working in Spark 3 i.e why the session is getting created on executor.
>>>> Another thing we tried is removing the df to rdd conversion just for
>>>> debug and it works in Spark 3.
>>>>
>>>> So, it might be something to do with df to rdd conversion or
>>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>>> But couldn't find the root cause.
>>>>
>>>> Regards,
>>>> Shrikant
>>>>
>>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> So call .setMaster("yarn"), per the error
>>>>>
>>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We are running it in cluster deploy mode with yarn.
>>>>>>
>>>>>> Regards,
>>>>>> Shrikant
>>>>>>
>>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>>
>>>>>>> The mater URL refers to as per java
>>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>>> according to where you want to run this
>>>>>>>
>>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to migrate one spark application from Spark 2.3 to
>>>>>>>> 3.0.1.
>>>>>>>>
>>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>>
>>>>>>>> object TestMain {
>>>>>>>>
>>>>>>>> val session =
>>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>>
>>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>>
>>>>>>>> import session.implicits._
>>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>>> x(0).toString).collect()
>>>>>>>> *println*(a.mkString("|"))
>>>>>>>>
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>>>>> below exception:
>>>>>>>>
>>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be
>>>>>>>> set in your configuration
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>>
>>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>>
>>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>>
>>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>>
>>>>>>>>
>>>>>>>> From the exception it appears that it tries to create spark session
>>>>>>>> on executor also in Spark 3 whereas its not created again on executor in
>>>>>>>> Spark 2.3.
>>>>>>>>
>>>>>>>> Can anyone help in identfying why there is this change in behavior?
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Shrikant
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shrikant Prasad
>>>>>>>>
>>>>>>> --
>>>>>> Regards,
>>>>>> Shrikant Prasad
>>>>>>
>>>>> --
>>>> Regards,
>>>> Shrikant Prasad
>>>>
>>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Sean Owen <sr...@gmail.com>.
Oh, it's because you are defining "spark" within your driver object, and
then it's getting serialized because you are trying to use TestMain methods
in your program.
This was never correct, but now it's an explicit error in Spark 3. The
session should not be a member variable.
On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad <sh...@gmail.com>
wrote:
> Please see these logs. The error is thrown in executor:
>
> 23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
> 0)
>
> java.lang.ExceptionInInitializerError
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
>
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:127)
>
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
>
> at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
>
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: org.apache.spark.SparkException: A master URL must be set in
> your configuration
>
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
>
> at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
>
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
>
> at scala.Option.getOrElse(Option.scala:189)
>
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
>
> at TestMain$.<init>(TestMain.scala:12)
>
> at TestMain$.<clinit>(TestMain.scala)
>
> On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> It's not running on the executor; that's not the issue. See your stack
>> trace, where it clearly happens in the driver.
>>
>> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
>> wrote:
>>
>>> Even if I set the master as yarn, it will not have access to rest of the
>>> spark confs. It will need spark.yarn.app.id.
>>>
>>> The main issue is if its working as it is in Spark 2.3 why its not
>>> working in Spark 3 i.e why the session is getting created on executor.
>>> Another thing we tried is removing the df to rdd conversion just for
>>> debug and it works in Spark 3.
>>>
>>> So, it might be something to do with df to rdd conversion or
>>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>>> But couldn't find the root cause.
>>>
>>> Regards,
>>> Shrikant
>>>
>>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> So call .setMaster("yarn"), per the error
>>>>
>>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
>>>> wrote:
>>>>
>>>>> We are running it in cluster deploy mode with yarn.
>>>>>
>>>>> Regards,
>>>>> Shrikant
>>>>>
>>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Can we see your Spark Configuration parameters ?
>>>>>>
>>>>>> The mater URL refers to as per java
>>>>>> new SparkConf()....setMaster("local[*]")
>>>>>> according to where you want to run this
>>>>>>
>>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>>>>>>
>>>>>>> The issue can be reproduced using below sample code:
>>>>>>>
>>>>>>> object TestMain {
>>>>>>>
>>>>>>> val session =
>>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>>
>>>>>>> def main(args: Array[String]): Unit = {
>>>>>>>
>>>>>>> import session.implicits._
>>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>>> x(0).toString).collect()
>>>>>>> *println*(a.mkString("|"))
>>>>>>>
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>>>> below exception:
>>>>>>>
>>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be set
>>>>>>> in your configuration
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>>
>>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>>
>>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>>
>>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>>
>>>>>>>
>>>>>>> From the exception it appears that it tries to create spark session
>>>>>>> on executor also in Spark 3 whereas its not created again on executor in
>>>>>>> Spark 2.3.
>>>>>>>
>>>>>>> Can anyone help in identfying why there is this change in behavior?
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Shrikant
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shrikant Prasad
>>>>>>>
>>>>>> --
>>>>> Regards,
>>>>> Shrikant Prasad
>>>>>
>>>> --
>>> Regards,
>>> Shrikant Prasad
>>>
>> --
> Regards,
> Shrikant Prasad
>
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
Please see these logs. The error is thrown in executor:
23/01/02 15:14:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: A master URL must be set in
your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:385)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
at TestMain$.<init>(TestMain.scala:12)
at TestMain$.<clinit>(TestMain.scala)
On Mon, 2 Jan 2023 at 8:29 PM, Sean Owen <sr...@gmail.com> wrote:
> It's not running on the executor; that's not the issue. See your stack
> trace, where it clearly happens in the driver.
>
> On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> Even if I set the master as yarn, it will not have access to rest of the
>> spark confs. It will need spark.yarn.app.id.
>>
>> The main issue is if its working as it is in Spark 2.3 why its not
>> working in Spark 3 i.e why the session is getting created on executor.
>> Another thing we tried is removing the df to rdd conversion just for
>> debug and it works in Spark 3.
>>
>> So, it might be something to do with df to rdd conversion or
>> serialization behavior change from Spark 2.3 to Spark 3.0 if there is any.
>> But couldn't find the root cause.
>>
>> Regards,
>> Shrikant
>>
>> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> So call .setMaster("yarn"), per the error
>>>
>>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
>>> wrote:
>>>
>>>> We are running it in cluster deploy mode with yarn.
>>>>
>>>> Regards,
>>>> Shrikant
>>>>
>>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can we see your Spark Configuration parameters ?
>>>>>
>>>>> The mater URL refers to as per java
>>>>> new SparkConf()....setMaster("local[*]")
>>>>> according to where you want to run this
>>>>>
>>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>>>>>
>>>>>> The issue can be reproduced using below sample code:
>>>>>>
>>>>>> object TestMain {
>>>>>>
>>>>>> val session =
>>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>>
>>>>>> def main(args: Array[String]): Unit = {
>>>>>>
>>>>>> import session.implicits._
>>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>>> x(0).toString).collect()
>>>>>> *println*(a.mkString("|"))
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>>> below exception:
>>>>>>
>>>>>> Caused by: org.apache.spark.SparkException: A master URL must be set
>>>>>> in your configuration
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>>
>>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>>
>>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>>
>>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>>
>>>>>>
>>>>>> From the exception it appears that it tries to create spark session
>>>>>> on executor also in Spark 3 whereas its not created again on executor in
>>>>>> Spark 2.3.
>>>>>>
>>>>>> Can anyone help in identfying why there is this change in behavior?
>>>>>>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Shrikant
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shrikant Prasad
>>>>>>
>>>>> --
>>>> Regards,
>>>> Shrikant Prasad
>>>>
>>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Sean Owen <sr...@gmail.com>.
It's not running on the executor; that's not the issue. See your stack
trace, where it clearly happens in the driver.
On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad <sh...@gmail.com>
wrote:
> Even if I set the master as yarn, it will not have access to rest of the
> spark confs. It will need spark.yarn.app.id.
>
> The main issue is if its working as it is in Spark 2.3 why its not working
> in Spark 3 i.e why the session is getting created on executor.
> Another thing we tried is removing the df to rdd conversion just for debug
> and it works in Spark 3.
>
> So, it might be something to do with df to rdd conversion or serialization
> behavior change from Spark 2.3 to Spark 3.0 if there is any. But couldn't
> find the root cause.
>
> Regards,
> Shrikant
>
> On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> So call .setMaster("yarn"), per the error
>>
>> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
>> wrote:
>>
>>> We are running it in cluster deploy mode with yarn.
>>>
>>> Regards,
>>> Shrikant
>>>
>>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>>> wrote:
>>>
>>>> Can we see your Spark Configuration parameters ?
>>>>
>>>> The mater URL refers to as per java
>>>> new SparkConf()....setMaster("local[*]")
>>>> according to where you want to run this
>>>>
>>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>>>>
>>>>> The issue can be reproduced using below sample code:
>>>>>
>>>>> object TestMain {
>>>>>
>>>>> val session =
>>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>>
>>>>> def main(args: Array[String]): Unit = {
>>>>>
>>>>> import session.implicits._
>>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>>> x(0).toString).collect()
>>>>> *println*(a.mkString("|"))
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with
>>>>> below exception:
>>>>>
>>>>> Caused by: org.apache.spark.SparkException: A master URL must be set
>>>>> in your configuration
>>>>>
>>>>> at
>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>>
>>>>> at
>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>>
>>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>>
>>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>>
>>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>>
>>>>>
>>>>> From the exception it appears that it tries to create spark session on
>>>>> executor also in Spark 3 whereas its not created again on executor in Spark
>>>>> 2.3.
>>>>>
>>>>> Can anyone help in identfying why there is this change in behavior?
>>>>>
>>>>> Thanks and Regards,
>>>>>
>>>>> Shrikant
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shrikant Prasad
>>>>>
>>>> --
>>> Regards,
>>> Shrikant Prasad
>>>
>> --
> Regards,
> Shrikant Prasad
>
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
Even if I set the master as yarn, it will not have access to rest of the
spark confs. It will need spark.yarn.app.id.
The main issue is if its working as it is in Spark 2.3 why its not working
in Spark 3 i.e why the session is getting created on executor.
Another thing we tried is removing the df to rdd conversion just for debug
and it works in Spark 3.
So, it might be something to do with df to rdd conversion or serialization
behavior change from Spark 2.3 to Spark 3.0 if there is any. But couldn't
find the root cause.
Regards,
Shrikant
On Mon, 2 Jan 2023 at 7:54 PM, Sean Owen <sr...@gmail.com> wrote:
> So call .setMaster("yarn"), per the error
>
> On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> We are running it in cluster deploy mode with yarn.
>>
>> Regards,
>> Shrikant
>>
>> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
>> wrote:
>>
>>> Can we see your Spark Configuration parameters ?
>>>
>>> The mater URL refers to as per java
>>> new SparkConf()....setMaster("local[*]")
>>> according to where you want to run this
>>>
>>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>>>
>>>> The issue can be reproduced using below sample code:
>>>>
>>>> object TestMain {
>>>>
>>>> val session =
>>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>>
>>>> def main(args: Array[String]): Unit = {
>>>>
>>>> import session.implicits._
>>>> val a = *session.*sparkContext.parallelize(*Array*
>>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>>> x(0).toString).collect()
>>>> *println*(a.mkString("|"))
>>>>
>>>> }
>>>> }
>>>>
>>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
>>>> exception:
>>>>
>>>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>>>> your configuration
>>>>
>>>> at
>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>>
>>>> at
>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>>
>>>> at
>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>>
>>>> at scala.Option.getOrElse(Option.scala:189)
>>>>
>>>> at
>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>>
>>>> at TestMain$.<init>(TestMain.scala:7)
>>>>
>>>> at TestMain$.<clinit>(TestMain.scala)
>>>>
>>>>
>>>> From the exception it appears that it tries to create spark session on
>>>> executor also in Spark 3 whereas its not created again on executor in Spark
>>>> 2.3.
>>>>
>>>> Can anyone help in identfying why there is this change in behavior?
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Shrikant
>>>>
>>>> --
>>>> Regards,
>>>> Shrikant Prasad
>>>>
>>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Sean Owen <sr...@gmail.com>.
So call .setMaster("yarn"), per the error
On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad <sh...@gmail.com>
wrote:
> We are running it in cluster deploy mode with yarn.
>
> Regards,
> Shrikant
>
> On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com>
> wrote:
>
>> Can we see your Spark Configuration parameters ?
>>
>> The mater URL refers to as per java
>> new SparkConf()....setMaster("local[*]")
>> according to where you want to run this
>>
>> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>>
>>> The issue can be reproduced using below sample code:
>>>
>>> object TestMain {
>>>
>>> val session =
>>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>>
>>> def main(args: Array[String]): Unit = {
>>>
>>> import session.implicits._
>>> val a = *session.*sparkContext.parallelize(*Array*
>>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>>> x(0).toString).collect()
>>> *println*(a.mkString("|"))
>>>
>>> }
>>> }
>>>
>>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
>>> exception:
>>>
>>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>>> your configuration
>>>
>>> at
>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>>
>>> at
>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>>
>>> at
>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>>
>>> at scala.Option.getOrElse(Option.scala:189)
>>>
>>> at
>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>>
>>> at TestMain$.<init>(TestMain.scala:7)
>>>
>>> at TestMain$.<clinit>(TestMain.scala)
>>>
>>>
>>> From the exception it appears that it tries to create spark session on
>>> executor also in Spark 3 whereas its not created again on executor in Spark
>>> 2.3.
>>>
>>> Can anyone help in identfying why there is this change in behavior?
>>>
>>> Thanks and Regards,
>>>
>>> Shrikant
>>>
>>> --
>>> Regards,
>>> Shrikant Prasad
>>>
>> --
> Regards,
> Shrikant Prasad
>
Re: Spark migration from 2.3 to 3.0.1
Posted by Shrikant Prasad <sh...@gmail.com>.
We are running it in cluster deploy mode with yarn.
Regards,
Shrikant
On Mon, 2 Jan 2023 at 6:15 PM, Stelios Philippou <st...@gmail.com> wrote:
> Can we see your Spark Configuration parameters ?
>
> The mater URL refers to as per java
> new SparkConf()....setMaster("local[*]")
> according to where you want to run this
>
> On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>>
>> The issue can be reproduced using below sample code:
>>
>> object TestMain {
>>
>> val session =
>> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>>
>> def main(args: Array[String]): Unit = {
>>
>> import session.implicits._
>> val a = *session.*sparkContext.parallelize(*Array*
>> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
>> x(0).toString).collect()
>> *println*(a.mkString("|"))
>>
>> }
>> }
>>
>> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
>> exception:
>>
>> Caused by: org.apache.spark.SparkException: A master URL must be set in
>> your configuration
>>
>> at
>> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>>
>> at
>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>>
>> at
>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>>
>> at scala.Option.getOrElse(Option.scala:189)
>>
>> at
>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>>
>> at TestMain$.<init>(TestMain.scala:7)
>>
>> at TestMain$.<clinit>(TestMain.scala)
>>
>>
>> From the exception it appears that it tries to create spark session on
>> executor also in Spark 3 whereas its not created again on executor in Spark
>> 2.3.
>>
>> Can anyone help in identfying why there is this change in behavior?
>>
>> Thanks and Regards,
>>
>> Shrikant
>>
>> --
>> Regards,
>> Shrikant Prasad
>>
> --
Regards,
Shrikant Prasad
Re: Spark migration from 2.3 to 3.0.1
Posted by Stelios Philippou <st...@gmail.com>.
Can we see your Spark Configuration parameters ?
The mater URL refers to as per java
new SparkConf()....setMaster("local[*]")
according to where you want to run this
On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <sh...@gmail.com> wrote:
> Hi,
>
> I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
>
> The issue can be reproduced using below sample code:
>
> object TestMain {
>
> val session =
> SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()
>
> def main(args: Array[String]): Unit = {
>
> import session.implicits._
> val a = *session.*sparkContext.parallelize(*Array*
> (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=>
> x(0).toString).collect()
> *println*(a.mkString("|"))
>
> }
> }
>
> It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
> exception:
>
> Caused by: org.apache.spark.SparkException: A master URL must be set in
> your configuration
>
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
>
> at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>
> at scala.Option.getOrElse(Option.scala:189)
>
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>
> at TestMain$.<init>(TestMain.scala:7)
>
> at TestMain$.<clinit>(TestMain.scala)
>
>
> From the exception it appears that it tries to create spark session on
> executor also in Spark 3 whereas its not created again on executor in Spark
> 2.3.
>
> Can anyone help in identfying why there is this change in behavior?
>
> Thanks and Regards,
>
> Shrikant
>
> --
> Regards,
> Shrikant Prasad
>