You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nick Shivhare <ni...@gmail.com> on 2021/10/14 07:13:00 UTC
apache-spark
Hi All,
We are facing an issue and would be thankful if anyone can help us on this
issue.
Environment: Spark, Kubernetes and Airflow.
Airflow is used to schedule job spark job over kubernetes.
We are using bash script which is using spark submit command to submit
spark jobs.
Issue:
We are submitting spark job through airflow in a *cluster mode*. However,
when the job is completed and executors are closed, airflow is not able to
schedule another job.
As per our investigation, we found that jobs are completed but spark submit
command in our script is not able to exit and continuously running with
following logs:
21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status for
spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
In order to confirm this issue is not from the airflow side, we tried to
kill spark submit command and it was able to schedule another job so our
observation is that somehow after completion of job spark-submit script is
not able to exit and still running.
FYI we have closed the spark session already in our code.
One of the weird observations was that it is running completely fine in
local mode and we are checking for client mode presently.
Would be thankful if you can guide us on this?
Thanks,
Shishir
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
When running a bash operator in airflow you can redirect the log and error
to a directory on the client host that is hosting airflow.
Example let us assume that your sh file is in /home/hduser/dba/bin/ and is
called resize_k8s.sh. You have a directory called run_logs for output from
the bash run. Your entry in your airflow should look like something below
# Resize gke cluster
create_command2 = '/home/hduser/dba/bin/resize_k8s.sh -N 3 >
/home/hduser/airflow/run_logs/t2_resize_k8s_`date +%Y%m%d_%H%M`.log 2>&1'
t2 = BashOperator(
task_id='t2_resize_k8s',
bash_command=create_command2,
dag=dag)
Both the log and errors are output to directory
/home/hduser/airflow/run_logs. The timestamped output will look like below
which will include any errors as well.
t2_resize_k8s_20211026_0925.log
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 22 Oct 2021 at 18:21, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Specifically,
>
> if [ $? != 0 ]
> then
>
> echo `date` " ""======= job failed, aborting! ======"
>
> exit 1
> fi
>
> exit 0
>
>
>
> On Fri, 22 Oct 2021 at 18:05, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Ok your problem seems to be the bash script that you are using your
>> spark-submit
>>
>> How are you exiting the bash script exit 0, exit 1? How are you
>> determining the jobs completed ok in the bash script?
>>
>> On Fri, 22 Oct 2021 at 14:58, Nick Shivhare <ni...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> Please find below my reply for the queries you have asked:
>>> 1. We are using AKS => Azure Kubernetes cluster
>>> 2. What operator in airflow are you using to submit your spark-submit
>>> for example BashOperator? => BashOperator, Through bashoperator we are
>>> calling a script which has a spark submit command.
>>> 3. Also have you tried to see what is going on within k8s driver?
>>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
>>> $1}'`
>>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>> *Reply:* Please find attached logs for the same. In our case the driver
>>> is completed.
>>> 4. Airflow cannot run two jobs in parallel when one is running
>>> Reply: We are running jobs in sequential order so it is not running jobs
>>> in parallel. It is waiting for the job to be completed so that it can
>>> trigger another job but the existing job is hanged it seems.
>>>
>>> 5. jstack of Spark Driver inside the pod to see if there is any
>>> non-daemon thread that blocks the DestroyJavaVM.
>>> Reply: Since the driver is completed there is no way we can login to the
>>> driver pod i think.
>>>
>>> NOTE: I am attaching a snapshot of showing driver completed and
>>> executors are deleted as job is finished.
>>>
>>> Summary: In our case our job is finished, if you check logs spark is
>>> shutdown gracefully, executors are deleted and driver is showing completed.
>>> Since we are submitting this job through bashoperator, spark submit is
>>> not exiting due to which our script is not exiting and our job is stuck in *cluster
>>> *mode. I am also attaching spark submit redirection logs written in
>>> spark-submit.log file.
>>> spark submit Command used inside bash script is attached as file in the
>>> mail
>>>
>>> Note: Please check "get_pods1.png" , you will find the last 2 lines
>>> highlighted with white colour are of our pods and
>>> "spark-pse-2021-07-02....." is showing status as completed.
>>>
>>> Thanks,
>>> Shishir
>>>
>>> On Thu, Oct 14, 2021 at 11:56 PM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> There seems to be two separate issues here
>>>>
>>>> 1) Airflow cannot run two jobs in parallel when one is running
>>>>
>>>> 2) Issue with spark on k8s
>>>>
>>>> Airflow will require 32GB or near to handle multiple jobs. Additionally
>>>> airflow comes standard on sql lite DB that can only handle a single job at
>>>> anytime. You can check airflow. cfg to find out about the database used.
>>>> Recommended multi-thread usage is to use PostgressDB.
>>>>
>>>> I have already made some comments about finding out why GKE is not
>>>> marking spark job as finished. I assume that you have sparkSession.stop()
>>>> at the end. Use kubectl to investigate the cause. Kubectl get pods - n
>>>> <NAMESPACE> will reveal what is still active. In a commercial env you
>>>> highly likely won’t have access to k8s cluster itself.
>>>>
>>>> HTH
>>>>
>>>> On Thu, 14 Oct 2021 at 15:16, Kent Yao <ya...@apache.org> wrote:
>>>>
>>>>> Which version of Spark are you using? It's better to provide this
>>>>> information with your question.
>>>>>
>>>>> Anyway, you can check the jstack of Spark Driver inside the pod to see
>>>>> if there is any non-daemon thread that blocks the DestroyJavaVM
>>>>>
>>>>> For example, for versions lower than 2.4.3/3.0.0, you may face such an
>>>>> issue that causes by an Okhttp Websocket non-daemon thread still
>>>>> active.
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> The phenomenon you described is very similar to what I said in this
>>>>> article - https://www.jianshu.com/p/fe5812ce6b27, in Mandarin.
>>>>> However, there are some related JIRA tickets in it that you can refer to.
>>>>>
>>>>> BR,
>>>>>
>>>>> Kent
>>>>>
>>>>> Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
>>>>>
>>>>>> Also have you tried to see what is going on within k8s driver?
>>>>>>
>>>>>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk
>>>>>> '{print $1}'`
>>>>>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>>>>>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Airflow is nothing but a new version of cron on linux with dag
>>>>>>> dependency. What operator in airflow are you using to submit your
>>>>>>> spark-submit for example BashOperator?
>>>>>>>
>>>>>>> Can you actually run the command outside of airflow by submitting
>>>>>>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>>> arise from relying on this email's technical content is explicitly
>>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>>> arising from such loss, damage or destruction.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <
>>>>>>> nickshivhares29@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We are facing an issue and would be thankful if anyone can help us
>>>>>>>> on this issue.
>>>>>>>> Environment: Spark, Kubernetes and Airflow.
>>>>>>>> Airflow is used to schedule job spark job over kubernetes.
>>>>>>>> We are using bash script which is using spark submit command to
>>>>>>>> submit spark jobs.
>>>>>>>> Issue:
>>>>>>>> We are submitting spark job through airflow in a *cluster mode*.
>>>>>>>> However, when the job is completed and executors are closed, airflow is not
>>>>>>>> able to schedule another job.
>>>>>>>>
>>>>>>>> As per our investigation, we found that jobs are completed but
>>>>>>>> spark submit command in our script is not able to exit and
>>>>>>>> continuously running with following logs:
>>>>>>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application
>>>>>>>> status for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>>>>>>
>>>>>>>> In order to confirm this issue is not from the airflow side, we
>>>>>>>> tried to kill spark submit command and it was able to schedule another job
>>>>>>>> so our observation is that somehow after completion of job spark-submit
>>>>>>>> script is not able to exit and still running.
>>>>>>>>
>>>>>>>> FYI we have closed the spark session already in our code.
>>>>>>>> One of the weird observations was that it is running completely
>>>>>>>> fine in local mode and we are checking for client mode presently.
>>>>>>>>
>>>>>>>>
>>>>>>>> Would be thankful if you can guide us on this?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Shishir
>>>>>>>>
>>>>>>> --
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>> --
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
> --
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Specifically,
if [ $? != 0 ]
then
echo `date` " ""======= job failed, aborting! ======"
exit 1
fi
exit 0
On Fri, 22 Oct 2021 at 18:05, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Ok your problem seems to be the bash script that you are using your
> spark-submit
>
> How are you exiting the bash script exit 0, exit 1? How are you
> determining the jobs completed ok in the bash script?
>
> On Fri, 22 Oct 2021 at 14:58, Nick Shivhare <ni...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Please find below my reply for the queries you have asked:
>> 1. We are using AKS => Azure Kubernetes cluster
>> 2. What operator in airflow are you using to submit your spark-submit for
>> example BashOperator? => BashOperator, Through bashoperator we are
>> calling a script which has a spark submit command.
>> 3. Also have you tried to see what is going on within k8s driver?
>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
>> $1}'`
>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>> *Reply:* Please find attached logs for the same. In our case the driver
>> is completed.
>> 4. Airflow cannot run two jobs in parallel when one is running
>> Reply: We are running jobs in sequential order so it is not running jobs
>> in parallel. It is waiting for the job to be completed so that it can
>> trigger another job but the existing job is hanged it seems.
>>
>> 5. jstack of Spark Driver inside the pod to see if there is any
>> non-daemon thread that blocks the DestroyJavaVM.
>> Reply: Since the driver is completed there is no way we can login to the
>> driver pod i think.
>>
>> NOTE: I am attaching a snapshot of showing driver completed and
>> executors are deleted as job is finished.
>>
>> Summary: In our case our job is finished, if you check logs spark is
>> shutdown gracefully, executors are deleted and driver is showing completed.
>> Since we are submitting this job through bashoperator, spark submit is
>> not exiting due to which our script is not exiting and our job is stuck in *cluster
>> *mode. I am also attaching spark submit redirection logs written in
>> spark-submit.log file.
>> spark submit Command used inside bash script is attached as file in the
>> mail
>>
>> Note: Please check "get_pods1.png" , you will find the last 2 lines
>> highlighted with white colour are of our pods and
>> "spark-pse-2021-07-02....." is showing status as completed.
>>
>> Thanks,
>> Shishir
>>
>> On Thu, Oct 14, 2021 at 11:56 PM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> There seems to be two separate issues here
>>>
>>> 1) Airflow cannot run two jobs in parallel when one is running
>>>
>>> 2) Issue with spark on k8s
>>>
>>> Airflow will require 32GB or near to handle multiple jobs. Additionally
>>> airflow comes standard on sql lite DB that can only handle a single job at
>>> anytime. You can check airflow. cfg to find out about the database used.
>>> Recommended multi-thread usage is to use PostgressDB.
>>>
>>> I have already made some comments about finding out why GKE is not
>>> marking spark job as finished. I assume that you have sparkSession.stop()
>>> at the end. Use kubectl to investigate the cause. Kubectl get pods - n
>>> <NAMESPACE> will reveal what is still active. In a commercial env you
>>> highly likely won’t have access to k8s cluster itself.
>>>
>>> HTH
>>>
>>> On Thu, 14 Oct 2021 at 15:16, Kent Yao <ya...@apache.org> wrote:
>>>
>>>> Which version of Spark are you using? It's better to provide this
>>>> information with your question.
>>>>
>>>> Anyway, you can check the jstack of Spark Driver inside the pod to see
>>>> if there is any non-daemon thread that blocks the DestroyJavaVM
>>>>
>>>> For example, for versions lower than 2.4.3/3.0.0, you may face such an
>>>> issue that causes by an Okhttp Websocket non-daemon thread still
>>>> active.
>>>>
>>>> [image: image.png]
>>>>
>>>> The phenomenon you described is very similar to what I said in this
>>>> article - https://www.jianshu.com/p/fe5812ce6b27, in Mandarin.
>>>> However, there are some related JIRA tickets in it that you can refer to.
>>>>
>>>> BR,
>>>>
>>>> Kent
>>>>
>>>> Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
>>>>
>>>>> Also have you tried to see what is going on within k8s driver?
>>>>>
>>>>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk
>>>>> '{print $1}'`
>>>>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>>>>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Airflow is nothing but a new version of cron on linux with dag
>>>>>> dependency. What operator in airflow are you using to submit your
>>>>>> spark-submit for example BashOperator?
>>>>>>
>>>>>> Can you actually run the command outside of airflow by submitting
>>>>>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <
>>>>>> nickshivhares29@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> We are facing an issue and would be thankful if anyone can help us
>>>>>>> on this issue.
>>>>>>> Environment: Spark, Kubernetes and Airflow.
>>>>>>> Airflow is used to schedule job spark job over kubernetes.
>>>>>>> We are using bash script which is using spark submit command to
>>>>>>> submit spark jobs.
>>>>>>> Issue:
>>>>>>> We are submitting spark job through airflow in a *cluster mode*.
>>>>>>> However, when the job is completed and executors are closed, airflow is not
>>>>>>> able to schedule another job.
>>>>>>>
>>>>>>> As per our investigation, we found that jobs are completed but spark
>>>>>>> submit command in our script is not able to exit and continuously running
>>>>>>> with following logs:
>>>>>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application
>>>>>>> status for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>>>>>
>>>>>>> In order to confirm this issue is not from the airflow side, we
>>>>>>> tried to kill spark submit command and it was able to schedule another job
>>>>>>> so our observation is that somehow after completion of job spark-submit
>>>>>>> script is not able to exit and still running.
>>>>>>>
>>>>>>> FYI we have closed the spark session already in our code.
>>>>>>> One of the weird observations was that it is running completely fine
>>>>>>> in local mode and we are checking for client mode presently.
>>>>>>>
>>>>>>>
>>>>>>> Would be thankful if you can guide us on this?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shishir
>>>>>>>
>>>>>> --
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>> --
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
--
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Ok your problem seems to be the bash script that you are using your
spark-submit
How are you exiting the bash script exit 0, exit 1? How are you determining
the jobs completed ok in the bash script?
On Fri, 22 Oct 2021 at 14:58, Nick Shivhare <ni...@gmail.com>
wrote:
> Hi All,
>
> Please find below my reply for the queries you have asked:
> 1. We are using AKS => Azure Kubernetes cluster
> 2. What operator in airflow are you using to submit your spark-submit for
> example BashOperator? => BashOperator, Through bashoperator we are
> calling a script which has a spark submit command.
> 3. Also have you tried to see what is going on within k8s driver?
> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
> $1}'`
> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
> *Reply:* Please find attached logs for the same. In our case the driver
> is completed.
> 4. Airflow cannot run two jobs in parallel when one is running
> Reply: We are running jobs in sequential order so it is not running jobs
> in parallel. It is waiting for the job to be completed so that it can
> trigger another job but the existing job is hanged it seems.
>
> 5. jstack of Spark Driver inside the pod to see if there is any
> non-daemon thread that blocks the DestroyJavaVM.
> Reply: Since the driver is completed there is no way we can login to the
> driver pod i think.
>
> NOTE: I am attaching a snapshot of showing driver completed and executors
> are deleted as job is finished.
>
> Summary: In our case our job is finished, if you check logs spark is
> shutdown gracefully, executors are deleted and driver is showing completed.
> Since we are submitting this job through bashoperator, spark submit is not
> exiting due to which our script is not exiting and our job is stuck in *cluster
> *mode. I am also attaching spark submit redirection logs written in
> spark-submit.log file.
> spark submit Command used inside bash script is attached as file in the
> mail
>
> Note: Please check "get_pods1.png" , you will find the last 2 lines
> highlighted with white colour are of our pods and
> "spark-pse-2021-07-02....." is showing status as completed.
>
> Thanks,
> Shishir
>
> On Thu, Oct 14, 2021 at 11:56 PM Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> There seems to be two separate issues here
>>
>> 1) Airflow cannot run two jobs in parallel when one is running
>>
>> 2) Issue with spark on k8s
>>
>> Airflow will require 32GB or near to handle multiple jobs. Additionally
>> airflow comes standard on sql lite DB that can only handle a single job at
>> anytime. You can check airflow. cfg to find out about the database used.
>> Recommended multi-thread usage is to use PostgressDB.
>>
>> I have already made some comments about finding out why GKE is not
>> marking spark job as finished. I assume that you have sparkSession.stop()
>> at the end. Use kubectl to investigate the cause. Kubectl get pods - n
>> <NAMESPACE> will reveal what is still active. In a commercial env you
>> highly likely won’t have access to k8s cluster itself.
>>
>> HTH
>>
>> On Thu, 14 Oct 2021 at 15:16, Kent Yao <ya...@apache.org> wrote:
>>
>>> Which version of Spark are you using? It's better to provide this
>>> information with your question.
>>>
>>> Anyway, you can check the jstack of Spark Driver inside the pod to see
>>> if there is any non-daemon thread that blocks the DestroyJavaVM
>>>
>>> For example, for versions lower than 2.4.3/3.0.0, you may face such an
>>> issue that causes by an Okhttp Websocket non-daemon thread still
>>> active.
>>>
>>> [image: image.png]
>>>
>>> The phenomenon you described is very similar to what I said in this
>>> article - https://www.jianshu.com/p/fe5812ce6b27, in Mandarin. However,
>>> there are some related JIRA tickets in it that you can refer to.
>>>
>>> BR,
>>>
>>> Kent
>>>
>>> Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
>>>
>>>> Also have you tried to see what is going on within k8s driver?
>>>>
>>>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk
>>>> '{print $1}'`
>>>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>>>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>>>
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Airflow is nothing but a new version of cron on linux with dag
>>>>> dependency. What operator in airflow are you using to submit your
>>>>> spark-submit for example BashOperator?
>>>>>
>>>>> Can you actually run the command outside of airflow by submitting
>>>>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We are facing an issue and would be thankful if anyone can help us
>>>>>> on this issue.
>>>>>> Environment: Spark, Kubernetes and Airflow.
>>>>>> Airflow is used to schedule job spark job over kubernetes.
>>>>>> We are using bash script which is using spark submit command to
>>>>>> submit spark jobs.
>>>>>> Issue:
>>>>>> We are submitting spark job through airflow in a *cluster mode*.
>>>>>> However, when the job is completed and executors are closed, airflow is not
>>>>>> able to schedule another job.
>>>>>>
>>>>>> As per our investigation, we found that jobs are completed but spark
>>>>>> submit command in our script is not able to exit and continuously running
>>>>>> with following logs:
>>>>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application
>>>>>> status for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>>>>
>>>>>> In order to confirm this issue is not from the airflow side, we tried
>>>>>> to kill spark submit command and it was able to schedule another job so our
>>>>>> observation is that somehow after completion of job spark-submit script is
>>>>>> not able to exit and still running.
>>>>>>
>>>>>> FYI we have closed the spark session already in our code.
>>>>>> One of the weird observations was that it is running completely fine
>>>>>> in local mode and we are checking for client mode presently.
>>>>>>
>>>>>>
>>>>>> Would be thankful if you can guide us on this?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Shishir
>>>>>>
>>>>> --
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
> --
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: apache-spark
Posted by Nick Shivhare <ni...@gmail.com>.
Hi All,
Please find below my reply for the queries you have asked:
1. We are using AKS => Azure Kubernetes cluster
2. What operator in airflow are you using to submit your spark-submit for
example BashOperator? => BashOperator, Through bashoperator we are calling
a script which has a spark submit command.
3. Also have you tried to see what is going on within k8s driver?
DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
$1}'`
kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
*Reply:* Please find attached logs for the same. In our case the driver is
completed.
4. Airflow cannot run two jobs in parallel when one is running
Reply: We are running jobs in sequential order so it is not running jobs in
parallel. It is waiting for the job to be completed so that it can trigger
another job but the existing job is hanged it seems.
5. jstack of Spark Driver inside the pod to see if there is any non-daemon
thread that blocks the DestroyJavaVM.
Reply: Since the driver is completed there is no way we can login to the
driver pod i think.
NOTE: I am attaching a snapshot of showing driver completed and executors
are deleted as job is finished.
Summary: In our case our job is finished, if you check logs spark is
shutdown gracefully, executors are deleted and driver is showing completed.
Since we are submitting this job through bashoperator, spark submit is not
exiting due to which our script is not exiting and our job is stuck in *cluster
*mode. I am also attaching spark submit redirection logs written in
spark-submit.log file.
spark submit Command used inside bash script is attached as file in the mail
Note: Please check "get_pods1.png" , you will find the last 2 lines
highlighted with white colour are of our pods and
"spark-pse-2021-07-02....." is showing status as completed.
Thanks,
Shishir
On Thu, Oct 14, 2021 at 11:56 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> There seems to be two separate issues here
>
> 1) Airflow cannot run two jobs in parallel when one is running
>
> 2) Issue with spark on k8s
>
> Airflow will require 32GB or near to handle multiple jobs. Additionally
> airflow comes standard on sql lite DB that can only handle a single job at
> anytime. You can check airflow. cfg to find out about the database used.
> Recommended multi-thread usage is to use PostgressDB.
>
> I have already made some comments about finding out why GKE is not marking
> spark job as finished. I assume that you have sparkSession.stop() at the
> end. Use kubectl to investigate the cause. Kubectl get pods - n <NAMESPACE>
> will reveal what is still active. In a commercial env you highly likely
> won’t have access to k8s cluster itself.
>
> HTH
>
> On Thu, 14 Oct 2021 at 15:16, Kent Yao <ya...@apache.org> wrote:
>
>> Which version of Spark are you using? It's better to provide this
>> information with your question.
>>
>> Anyway, you can check the jstack of Spark Driver inside the pod to see if
>> there is any non-daemon thread that blocks the DestroyJavaVM
>>
>> For example, for versions lower than 2.4.3/3.0.0, you may face such an
>> issue that causes by an Okhttp Websocket non-daemon thread still active.
>>
>> [image: image.png]
>>
>> The phenomenon you described is very similar to what I said in this
>> article - https://www.jianshu.com/p/fe5812ce6b27, in Mandarin. However,
>> there are some related JIRA tickets in it that you can refer to.
>>
>> BR,
>>
>> Kent
>>
>> Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
>>
>>> Also have you tried to see what is going on within k8s driver?
>>>
>>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
>>> $1}'`
>>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>>
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Airflow is nothing but a new version of cron on linux with dag
>>>> dependency. What operator in airflow are you using to submit your
>>>> spark-submit for example BashOperator?
>>>>
>>>> Can you actually run the command outside of airflow by submitting
>>>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>>>
>>>> HTH
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We are facing an issue and would be thankful if anyone can help us on
>>>>> this issue.
>>>>> Environment: Spark, Kubernetes and Airflow.
>>>>> Airflow is used to schedule job spark job over kubernetes.
>>>>> We are using bash script which is using spark submit command to submit
>>>>> spark jobs.
>>>>> Issue:
>>>>> We are submitting spark job through airflow in a *cluster mode*.
>>>>> However, when the job is completed and executors are closed, airflow is not
>>>>> able to schedule another job.
>>>>>
>>>>> As per our investigation, we found that jobs are completed but spark
>>>>> submit command in our script is not able to exit and continuously running
>>>>> with following logs:
>>>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status
>>>>> for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>>>
>>>>> In order to confirm this issue is not from the airflow side, we tried
>>>>> to kill spark submit command and it was able to schedule another job so our
>>>>> observation is that somehow after completion of job spark-submit script is
>>>>> not able to exit and still running.
>>>>>
>>>>> FYI we have closed the spark session already in our code.
>>>>> One of the weird observations was that it is running completely fine
>>>>> in local mode and we are checking for client mode presently.
>>>>>
>>>>>
>>>>> Would be thankful if you can guide us on this?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Shishir
>>>>>
>>>> --
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
There seems to be two separate issues here
1) Airflow cannot run two jobs in parallel when one is running
2) Issue with spark on k8s
Airflow will require 32GB or near to handle multiple jobs. Additionally
airflow comes standard on sql lite DB that can only handle a single job at
anytime. You can check airflow. cfg to find out about the database used.
Recommended multi-thread usage is to use PostgressDB.
I have already made some comments about finding out why GKE is not marking
spark job as finished. I assume that you have sparkSession.stop() at the
end. Use kubectl to investigate the cause. Kubectl get pods - n <NAMESPACE>
will reveal what is still active. In a commercial env you highly likely
won’t have access to k8s cluster itself.
HTH
On Thu, 14 Oct 2021 at 15:16, Kent Yao <ya...@apache.org> wrote:
> Which version of Spark are you using? It's better to provide this
> information with your question.
>
> Anyway, you can check the jstack of Spark Driver inside the pod to see if
> there is any non-daemon thread that blocks the DestroyJavaVM
>
> For example, for versions lower than 2.4.3/3.0.0, you may face such an
> issue that causes by an Okhttp Websocket non-daemon thread still active.
>
> [image: image.png]
>
> The phenomenon you described is very similar to what I said in this
> article - https://www.jianshu.com/p/fe5812ce6b27, in Mandarin. However,
> there are some related JIRA tickets in it that you can refer to.
>
> BR,
>
> Kent
>
> Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
>
>> Also have you tried to see what is going on within k8s driver?
>>
>> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
>> $1}'`
>> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
>> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>>
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Airflow is nothing but a new version of cron on linux with dag
>>> dependency. What operator in airflow are you using to submit your
>>> spark-submit for example BashOperator?
>>>
>>> Can you actually run the command outside of airflow by submitting
>>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>>
>>> HTH
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> We are facing an issue and would be thankful if anyone can help us on
>>>> this issue.
>>>> Environment: Spark, Kubernetes and Airflow.
>>>> Airflow is used to schedule job spark job over kubernetes.
>>>> We are using bash script which is using spark submit command to submit
>>>> spark jobs.
>>>> Issue:
>>>> We are submitting spark job through airflow in a *cluster mode*.
>>>> However, when the job is completed and executors are closed, airflow is not
>>>> able to schedule another job.
>>>>
>>>> As per our investigation, we found that jobs are completed but spark
>>>> submit command in our script is not able to exit and continuously running
>>>> with following logs:
>>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status
>>>> for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>>
>>>> In order to confirm this issue is not from the airflow side, we tried
>>>> to kill spark submit command and it was able to schedule another job so our
>>>> observation is that somehow after completion of job spark-submit script is
>>>> not able to exit and still running.
>>>>
>>>> FYI we have closed the spark session already in our code.
>>>> One of the weird observations was that it is running completely fine in
>>>> local mode and we are checking for client mode presently.
>>>>
>>>>
>>>> Would be thankful if you can guide us on this?
>>>>
>>>>
>>>> Thanks,
>>>> Shishir
>>>>
>>> --
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: apache-spark
Posted by Kent Yao <ya...@apache.org>.
Which version of Spark are you using? It's better to provide this
information with your question.
Anyway, you can check the jstack of Spark Driver inside the pod to see if
there is any non-daemon thread that blocks the DestroyJavaVM
For example, for versions lower than 2.4.3/3.0.0, you may face such an
issue that causes by an Okhttp Websocket non-daemon thread still active.
[image: image.png]
The phenomenon you described is very similar to what I said in this article
- https://www.jianshu.com/p/fe5812ce6b27, in Mandarin. However, there are
some related JIRA tickets in it that you can refer to.
BR,
Kent
Mich Talebzadeh <mi...@gmail.com> 于2021年10月14日周四 下午9:53写道：
> Also have you tried to see what is going on within k8s driver?
>
> DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
> $1}'`
> kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
> kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
>
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Airflow is nothing but a new version of cron on linux with dag
>> dependency. What operator in airflow are you using to submit your
>> spark-submit for example BashOperator?
>>
>> Can you actually run the command outside of airflow by submitting
>> spark-submit to K8s cluster? Is that GKE cluster or something else?
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We are facing an issue and would be thankful if anyone can help us on
>>> this issue.
>>> Environment: Spark, Kubernetes and Airflow.
>>> Airflow is used to schedule job spark job over kubernetes.
>>> We are using bash script which is using spark submit command to submit
>>> spark jobs.
>>> Issue:
>>> We are submitting spark job through airflow in a *cluster mode*.
>>> However, when the job is completed and executors are closed, airflow is not
>>> able to schedule another job.
>>>
>>> As per our investigation, we found that jobs are completed but spark
>>> submit command in our script is not able to exit and continuously running
>>> with following logs:
>>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status
>>> for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>>
>>> In order to confirm this issue is not from the airflow side, we tried to
>>> kill spark submit command and it was able to schedule another job so our
>>> observation is that somehow after completion of job spark-submit script is
>>> not able to exit and still running.
>>>
>>> FYI we have closed the spark session already in our code.
>>> One of the weird observations was that it is running completely fine in
>>> local mode and we are checking for client mode presently.
>>>
>>>
>>> Would be thankful if you can guide us on this?
>>>
>>>
>>> Thanks,
>>> Shishir
>>>
>>
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Also have you tried to see what is going on within k8s driver?
DRIVER_POD_NAME=`kubectl get pods -n $NAMESPACE |grep driver|awk '{print
$1}'`
kubectl describe pod $DRIVER_POD_NAME -n $NAMESPACE
kubectl logs $DRIVER_POD_NAME -n $NAMESPACE
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 14 Oct 2021 at 14:13, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> Airflow is nothing but a new version of cron on linux with dag dependency.
> What operator in airflow are you using to submit your spark-submit for
> example BashOperator?
>
> Can you actually run the command outside of airflow by submitting
> spark-submit to K8s cluster? Is that GKE cluster or something else?
>
> HTH
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We are facing an issue and would be thankful if anyone can help us on
>> this issue.
>> Environment: Spark, Kubernetes and Airflow.
>> Airflow is used to schedule job spark job over kubernetes.
>> We are using bash script which is using spark submit command to submit
>> spark jobs.
>> Issue:
>> We are submitting spark job through airflow in a *cluster mode*.
>> However, when the job is completed and executors are closed, airflow is not
>> able to schedule another job.
>>
>> As per our investigation, we found that jobs are completed but spark
>> submit command in our script is not able to exit and continuously running
>> with following logs:
>> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status
>> for spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>>
>> In order to confirm this issue is not from the airflow side, we tried to
>> kill spark submit command and it was able to schedule another job so our
>> observation is that somehow after completion of job spark-submit script is
>> not able to exit and still running.
>>
>> FYI we have closed the spark session already in our code.
>> One of the weird observations was that it is running completely fine in
>> local mode and we are checking for client mode presently.
>>
>>
>> Would be thankful if you can guide us on this?
>>
>>
>> Thanks,
>> Shishir
>>
>
Re: apache-spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Airflow is nothing but a new version of cron on linux with dag dependency.
What operator in airflow are you using to submit your spark-submit for
example BashOperator?
Can you actually run the command outside of airflow by submitting
spark-submit to K8s cluster? Is that GKE cluster or something else?
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <ni...@gmail.com>
wrote:
> Hi All,
>
> We are facing an issue and would be thankful if anyone can help us on
> this issue.
> Environment: Spark, Kubernetes and Airflow.
> Airflow is used to schedule job spark job over kubernetes.
> We are using bash script which is using spark submit command to submit
> spark jobs.
> Issue:
> We are submitting spark job through airflow in a *cluster mode*. However,
> when the job is completed and executors are closed, airflow is not able to
> schedule another job.
>
> As per our investigation, we found that jobs are completed but spark
> submit command in our script is not able to exit and continuously running
> with following logs:
> 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status for
> spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running)
>
> In order to confirm this issue is not from the airflow side, we tried to
> kill spark submit command and it was able to schedule another job so our
> observation is that somehow after completion of job spark-submit script is
> not able to exit and still running.
>
> FYI we have closed the spark session already in our code.
> One of the weird observations was that it is running completely fine in
> local mode and we are checking for client mode presently.
>
>
> Would be thankful if you can guide us on this?
>
>
> Thanks,
> Shishir
>