Spark SQL job block when use hive udf from_unixtime

["Dai, Kevin" <yu...@ebay.com>]

Hi, there

When I use hive udf from_unixtime with the HiveContext, the job block and the log is as follow:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

But if I replace it with my own udf, it works.

My data is from HBase
Whether I cache the data in memory or save it as the parquet file and load it later, the job still blocks


How can I fix it?



Thanks,
Kevin.

Re: Spark SQL job block when use hive udf from_unixtime

[Cheng Lian <li...@gmail.com>]

Could you please provide a complete stacktrace? Also it would be good if 
you can share your hive-site.xml as well.

On 12/23/14 4:42 PM, Dai, Kevin wrote:
>
> Hi, there
>
> When I use hive udf from_unixtime with the HiveContext, the job block 
> and the log is as follow:
>
> sun.misc.Unsafe.park(Native Method)
>
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
>
> But if I replace it with my own udf, it works.
>
> My data is from HBase
>
> Whether I cache the data in memory or save it as the parquet file and 
> load it later, the job still blocks
>
> How can I fix it?
>
> Thanks,
>
> Kevin.
>