Re: Spark on K8s , some applications ended ungracefully

[Martin Grigorov <mg...@apache.org>]

Hi,

On Thu, Mar 31, 2022 at 4:18 PM Pralabh Kumar <pr...@gmail.com>
wrote:

> Hi Spark Team
>
> Some of my spark applications on K8s ended with the below error . These
> applications though completed successfully (as per the event log
> SparkListenerApplicationEnd event at the end)
> stil have even files with .inprogress. This causes the application to be
> shown as inprogress in SHS.
>
> Spark v : 3.0.1
>

I'd suggest you to try with newer version, e.g. 3.2.1 or even one built
from branch-3.3.



>
>
>
> 22/03/31 08:33:34 WARN ShutdownHookManager: ShutdownHook '$anon$2'
> timeout, java.util.concurrent.TimeoutException
>
> java.util.concurrent.TimeoutException
>
>                 at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>
>                 at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68)
>
> 22/03/31 08:33:34 WARN SparkContext: Ignoring Exception while stopping
> SparkContext from shutdown hook
>
> java.lang.InterruptedException
>
>                 at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>
>                 at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
>
>                 at
> org.apache.spark.util.ThreadUtils$.shutdown(ThreadUtils.scala:348)
>
>
>
>
>
> Please let me know if there is a solution for it ..
>
> Regards
>
> Pralabh Kumar
>