Spark Standalone cluster job to connect Hbase is Stuck

[sudhir patil <sp...@gmail.com>]

Spark job on Standalone cluster is Stuck, shows no logs after
"util.AkkaUtils: Connecting to HeartbeatReceiver" on worker nodes and
"storage.BlockmanagerInfo: Added broadcast..." on client driver side.

Would be great, if you could clarify any of these ( or better all of these
:)
1. Did anyone see similar issue? or any clues on what could be the reason?
2. How do you i increase debug or log level? to see whats actually
happening?
3. Any clues or links on how to use kerborised Hbase in spark standalone?

Re: Spark Standalone cluster job to connect Hbase is Stuck

[Ted Yu <yu...@gmail.com>]

>From your first email, it seems that you don't observer output from hbase
client.
Spark 1.2 was quite old, missing fixes for log4j such as SPARK-9826

Can you change the following line in HBase's conf/log4j.properties
from:
log4j.logger.org.apache.hadoop.hbase=INFO

to:
log4j.logger.org.apache.hadoop.hbase=DEBUG

on client and server sides (restart servers for the above to take effect)
to see if you would have more clue ?

On Mon, Feb 1, 2016 at 4:44 PM, sudhir patil <sp...@gmail.com>
wrote:

> Thanks Ted for quick reply.
>
> I am using spark 1.2, exporting Hbase conf directory containing
> hbase-site.xml in HADOOP_CLASSPATH & SPARK_CLASSPATH. Do i need to do
> anything else?
>
> Issues in connecting to kerberos Hbase through spark yarn cluster is fixed
> spark 1.4+, so i am trying if it works in spark stand alone mode in spark
> 1.2, as i cannot upgrade cluster.
> https://issues.apache.org/jira/browse/SPARK-6918
>
>
>
> On Tue, Feb 2, 2016 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Is the hbase-site.xml on the classpath of the worker nodes ?
>>
>> Which Spark release are you using ?
>>
>> Cheers
>>
>> On Mon, Feb 1, 2016 at 4:25 PM, sudhir patil <sp...@gmail.com>
>> wrote:
>>
>>> Spark job on Standalone cluster is Stuck, shows no logs after
>>> "util.AkkaUtils: Connecting to HeartbeatReceiver" on worker nodes and
>>> "storage.BlockmanagerInfo: Added broadcast..." on client driver side.
>>>
>>> Would be great, if you could clarify any of these ( or better all of
>>> these :)
>>> 1. Did anyone see similar issue? or any clues on what could be the
>>> reason?
>>> 2. How do you i increase debug or log level? to see whats actually
>>> happening?
>>> 3. Any clues or links on how to use kerborised Hbase in spark standalone?
>>>
>>
>>
>

Re: Spark Standalone cluster job to connect Hbase is Stuck

[sudhir patil <sp...@gmail.com>]

Thanks Ted for quick reply.

I am using spark 1.2, exporting Hbase conf directory containing
hbase-site.xml in HADOOP_CLASSPATH & SPARK_CLASSPATH. Do i need to do
anything else?

Issues in connecting to kerberos Hbase through spark yarn cluster is fixed
spark 1.4+, so i am trying if it works in spark stand alone mode in spark
1.2, as i cannot upgrade cluster.
https://issues.apache.org/jira/browse/SPARK-6918



On Tue, Feb 2, 2016 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:

> Is the hbase-site.xml on the classpath of the worker nodes ?
>
> Which Spark release are you using ?
>
> Cheers
>
> On Mon, Feb 1, 2016 at 4:25 PM, sudhir patil <sp...@gmail.com>
> wrote:
>
>> Spark job on Standalone cluster is Stuck, shows no logs after
>> "util.AkkaUtils: Connecting to HeartbeatReceiver" on worker nodes and
>> "storage.BlockmanagerInfo: Added broadcast..." on client driver side.
>>
>> Would be great, if you could clarify any of these ( or better all of
>> these :)
>> 1. Did anyone see similar issue? or any clues on what could be the reason?
>> 2. How do you i increase debug or log level? to see whats actually
>> happening?
>> 3. Any clues or links on how to use kerborised Hbase in spark standalone?
>>
>
>

Re: Spark Standalone cluster job to connect Hbase is Stuck

[Ted Yu <yu...@gmail.com>]

Is the hbase-site.xml on the classpath of the worker nodes ?

Which Spark release are you using ?

Cheers

On Mon, Feb 1, 2016 at 4:25 PM, sudhir patil <sp...@gmail.com>
wrote:

> Spark job on Standalone cluster is Stuck, shows no logs after
> "util.AkkaUtils: Connecting to HeartbeatReceiver" on worker nodes and
> "storage.BlockmanagerInfo: Added broadcast..." on client driver side.
>
> Would be great, if you could clarify any of these ( or better all of these
> :)
> 1. Did anyone see similar issue? or any clues on what could be the reason?
> 2. How do you i increase debug or log level? to see whats actually
> happening?
> 3. Any clues or links on how to use kerborised Hbase in spark standalone?
>