You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by yxl040840219 <yx...@126.com> on 2021/03/11 12:05:56 UTC
spark on k8s driver pod exception
when run the code in k8s , driver pod throw AnalysisException , but the spark-submit log still running , then how to get the exception and stop pods ?
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
.groupBy("key").agg(count("value1").as("cnt"))
df.show()
spark.stop()
bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Re: spark on k8s driver pod exception
Posted by Attila Zsolt Piros <pi...@gmail.com>.
Sure, that is expected, see the "How it works" section in "Running Spark on
Kubernetes" page
<https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#how-it-works>,
quote:
When the application completes, the executor pods terminate and are cleaned
> up, but the driver pod persists logs and remains in “completed” state in
> the Kubernetes API until it’s eventually garbage collected or manually
> cleaned up.
On Mon, Mar 15, 2021 at 8:45 AM 040840219 <yx...@126.com> wrote:
>
> when driver pod throws exception , driver pod still running ？
>
> kubectl logs wordcount-e3141c7834d3dd68-driver
>
> 21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to
> 'value1
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
> resolve '`value1`' given input columns: [key, value];
> 'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
> +- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
> +- Project [value#1 AS id#4]
> +- LocalRelation [value#1]
>
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
>
> kubectl get pods wordcount-e3141c7834d3dd68-driver
>
> NAME READY STATUS RESTARTS AGE
> wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s
>
> On 03/12/2021 05:42，Attila Zsolt Piros<pi...@gmail.com>
> <pi...@gmail.com> wrote：
>
> > but the spark-submit log still running
>
> Set the "spark.kubernetes.submission.waitAppCompletion" config to false to
> change that. As the doc says:
>
> "spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether
> to wait for the application to finish before exiting the launcher process.
> When changed to false, the launcher has a "fire-and-forget" behavior when
> launching the Spark job.
>
> On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <
> piros.attila.zsolt@gmail.com> wrote:
>
>>
>> For getting the logs please read Accessing Logs
>> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#accessing-logs> part
>> of the *Running Spark on Kubernetes* page.
>>
>> For stopping and generic management of the spark application please read
>> the Spark Application Management
>> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#spark-application-management>,
>> where you find the example:
>>
>> $ spark-submit --kill spark:spark-pi* --master k8s://https://192.168.2.8:8443
>>
>>
>>
>> On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yx...@126.com>
>> wrote:
>>
>>>
>>>
>>>
>>> when run the code in k8s , driver pod throw AnalysisException , but
>>> the spark-submit log still running , then how to get the exception and
>>> stop pods ?
>>>
>>> val spark = SparkSession.builder().getOrCreate()
>>> import spark.implicits._
>>> val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key",
>>> "id%10 as value")
>>> .groupBy("key").agg(count("value1").as("cnt"))
>>> df.show()
>>> spark.stop()
>>>
>>> bin/spark-submit \
>>> --master k8s://https://localhost:9443 \
>>> --deploy-mode cluster \
>>> --name wordcount \
>>> --class k8s.WordCount \
>>> --conf spark.kubernetes.container.image=rspark:v3.1.1 \
>>> --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
>>> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>> --conf
>>> spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
>>> /data/spark-example-1.0.0.jar
>>>
>>
Re: spark on k8s driver pod exception
Posted by 040840219 <yx...@126.com>.
when driver pod throws exception , driver pod still running ？
kubectl logs wordcount-e3141c7834d3dd68-driver
21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`value1`' given input columns: [key, value];
'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
+- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
+- Project [value#1 AS id#4]
+- LocalRelation [value#1]
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
kubectl get pods wordcount-e3141c7834d3dd68-driver
NAME READY STATUS RESTARTS AGE
wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s
On 03/12/2021 05:42，Attila Zsolt Piros<pi...@gmail.com> wrote：
> but the spark-submit log still running
Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says:
"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job.
On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <pi...@gmail.com> wrote:
For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page.
For stopping and generic management of the spark application please read the Spark Application Management, where you find the example:
$ spark-submit --kill spark:spark-pi*--master k8s://https://192.168.2.8:8443
On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yx...@126.com> wrote:
when run the code in k8s , driver pod throw AnalysisException , but the spark-submit log still running , then how to get the exception and stop pods ?
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
.groupBy("key").agg(count("value1").as("cnt"))
df.show()
spark.stop()
bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Re: spark on k8s driver pod exception
Posted by Attila Zsolt Piros <pi...@gmail.com>.
> but the spark-submit log still running
Set the "spark.kubernetes.submission.waitAppCompletion" config to false to
change that. As the doc says:
"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether
to wait for the application to finish before exiting the launcher process.
When changed to false, the launcher has a "fire-and-forget" behavior when
launching the Spark job.
On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <
piros.attila.zsolt@gmail.com> wrote:
>
> For getting the logs please read Accessing Logs
> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#accessing-logs> part
> of the *Running Spark on Kubernetes* page.
>
> For stopping and generic management of the spark application please read
> the Spark Application Management
> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#spark-application-management>,
> where you find the example:
>
> $ spark-submit --kill spark:spark-pi* --master k8s://https://192.168.2.8:8443
>
>
>
> On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yx...@126.com> wrote:
>
>>
>>
>>
>> when run the code in k8s , driver pod throw AnalysisException , but the
>> spark-submit log still running , then how to get the exception and stop
>> pods ?
>>
>> val spark = SparkSession.builder().getOrCreate()
>> import spark.implicits._
>> val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key",
>> "id%10 as value")
>> .groupBy("key").agg(count("value1").as("cnt"))
>> df.show()
>> spark.stop()
>>
>> bin/spark-submit \
>> --master k8s://https://localhost:9443 \
>> --deploy-mode cluster \
>> --name wordcount \
>> --class k8s.WordCount \
>> --conf spark.kubernetes.container.image=rspark:v3.1.1 \
>> --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
>> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>> --conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark
>> \
>> /data/spark-example-1.0.0.jar
>>
>
Re: spark on k8s driver pod exception
Posted by Attila Zsolt Piros <pi...@gmail.com>.
For getting the logs please read Accessing Logs
<https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#accessing-logs>
part
of the *Running Spark on Kubernetes* page.
For stopping and generic management of the spark application please read
the Spark Application Management
<https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#spark-application-management>,
where you find the example:
$ spark-submit --kill spark:spark-pi* --master k8s://https://192.168.2.8:8443
On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yx...@126.com> wrote:
>
>
>
> when run the code in k8s , driver pod throw AnalysisException , but the
> spark-submit log still running , then how to get the exception and stop
> pods ?
>
> val spark = SparkSession.builder().getOrCreate()
> import spark.implicits._
> val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key",
> "id%10 as value")
> .groupBy("key").agg(count("value1").as("cnt"))
> df.show()
> spark.stop()
>
> bin/spark-submit \
> --master k8s://https://localhost:9443 \
> --deploy-mode cluster \
> --name wordcount \
> --class k8s.WordCount \
> --conf spark.kubernetes.container.image=rspark:v3.1.1 \
> --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
> /data/spark-example-1.0.0.jar
>