[jira] [Commented] (KYLIN-4099) Using no blocking RDD unpersist in spark cubing job

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888527#comment-16888527 ] 

ASF GitHub Bot commented on KYLIN-4099:
---------------------------------------

lshmouse commented on pull request #762: KYLIN-4099: Using no blocking RDD unpersist in spark cubing job
URL: https://github.com/apache/kylin/pull/762
 
 
   See: https://issues.apache.org/jira/browse/KYLIN-4099
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Using no blocking RDD unpersist in spark cubing job 
> ----------------------------------------------------
>
>                 Key: KYLIN-4099
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4099
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>            Priority: Major
>
> By default, the unpersist operation of RDD in spark is blocking which may cost a lot time and
> some times it may failed for some spark executors lost. 
> We can set blocking false to improve it.
> {code:java}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> scala.concurrent.Await$.result(package.scala:190)
> org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
> org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
> org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
> org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
> org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)