You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by yaoxiaohua <ya...@outlook.com> on 2015/12/22 12:30:04 UTC
Client session timed out, have not heard from server in
Hi,
I encounter a similar question, spark1.4
Master2 run some days , then give a timeout exception, then shutdown.
I found a bug :
https://issues.apache.org/jira/browse/SPARK-9629
INFO ClientCnxn: Client session timed out, have not heard from server in
40015ms for sessionid 0x351c416297a145a, closing socket connection and
attempting reconnect
could you tell me what do you do for this?
Best Regards,
Evan
Re: Client session timed out, have not heard from server in
Posted by Yash Sharma <ya...@gmail.com>.
Evan could you also share more logs on the error. Probably paste here or
in pastebin.
Also check zookeeper logs in case you find anything.
- Thanks, via mobile, excuse brevity.
On Dec 22, 2015 6:01 PM, "Dirceu Semighini Filho" <
dirceu.semighini@gmail.com> wrote:
> Hi Yash,
> I've experienced this behavior here when the process freeze in a worker.
> This mainly happen, in my case, when the worker memory was full and the
> java GC wasn't able to free memory for the process.
> Try to search for outofmemory error in your worker logs.
>
> Regards,
> Dirceu
>
> 2015-12-22 10:26 GMT-02:00 yaoxiaohua <ya...@outlook.com>:
>
>> Thanks for your reply.
>>
>> I find spark-env.sh :
>>
>> SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300
>> -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true
>> -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs
>> -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash
>> -Dspark.yarn.max.executor.failures=99999 -Dspark.worker.timeout=300"
>>
>>
>>
>> I just find log like this:
>>
>>
>> INFO ClientCnxn: Client session timed out, have not heard from server in
>> 40015ms for sessionid 0x351c416297a145a, closing socket connection and
>> attempting reconnect
>>
>> Before spark2 master process shut down.
>>
>> I don’t see any zookeeper timeout setting .
>>
>>
>>
>> Best
>>
>>
>>
>> *From:* Yash Sharma [mailto:yash360@gmail.com]
>> *Sent:* 2015年12月22日 19:55
>> *To:* yaoxiaohua
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Client session timed out, have not heard from server in
>>
>>
>>
>> Hi Evan,
>> SPARK-9629 referred to connection issues with zookeeper. Could you check
>> if its working fine in your setup.
>>
>> Also please share other error logs you might be getting.
>>
>> - Thanks, via mobile, excuse brevity.
>>
>> On Dec 22, 2015 5:00 PM, "yaoxiaohua" <ya...@outlook.com> wrote:
>>
>> Hi,
>>
>> I encounter a similar question, spark1.4
>>
>> Master2 run some days , then give a timeout exception, then shutdown.
>>
>> I found a bug :
>>
>> https://issues.apache.org/jira/browse/SPARK-9629
>>
>>
>>
>>
>> INFO ClientCnxn: Client session timed out, have not heard from server in
>> 40015ms for sessionid 0x351c416297a145a, closing socket connection and
>> attempting reconnect
>>
>>
>>
>>
>>
>> could you tell me what do you do for this?
>>
>>
>>
>> Best Regards,
>>
>> Evan
>>
>
>
Re: Client session timed out, have not heard from server in
Posted by Dirceu Semighini Filho <di...@gmail.com>.
Hi Yash,
I've experienced this behavior here when the process freeze in a worker.
This mainly happen, in my case, when the worker memory was full and the
java GC wasn't able to free memory for the process.
Try to search for outofmemory error in your worker logs.
Regards,
Dirceu
2015-12-22 10:26 GMT-02:00 yaoxiaohua <ya...@outlook.com>:
> Thanks for your reply.
>
> I find spark-env.sh :
>
> SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300
> -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true
> -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs
> -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash
> -Dspark.yarn.max.executor.failures=99999 -Dspark.worker.timeout=300"
>
>
>
> I just find log like this:
>
>
> INFO ClientCnxn: Client session timed out, have not heard from server in
> 40015ms for sessionid 0x351c416297a145a, closing socket connection and
> attempting reconnect
>
> Before spark2 master process shut down.
>
> I don’t see any zookeeper timeout setting .
>
>
>
> Best
>
>
>
> *From:* Yash Sharma [mailto:yash360@gmail.com]
> *Sent:* 2015年12月22日 19:55
> *To:* yaoxiaohua
> *Cc:* user@spark.apache.org
> *Subject:* Re: Client session timed out, have not heard from server in
>
>
>
> Hi Evan,
> SPARK-9629 referred to connection issues with zookeeper. Could you check
> if its working fine in your setup.
>
> Also please share other error logs you might be getting.
>
> - Thanks, via mobile, excuse brevity.
>
> On Dec 22, 2015 5:00 PM, "yaoxiaohua" <ya...@outlook.com> wrote:
>
> Hi,
>
> I encounter a similar question, spark1.4
>
> Master2 run some days , then give a timeout exception, then shutdown.
>
> I found a bug :
>
> https://issues.apache.org/jira/browse/SPARK-9629
>
>
>
>
> INFO ClientCnxn: Client session timed out, have not heard from server in
> 40015ms for sessionid 0x351c416297a145a, closing socket connection and
> attempting reconnect
>
>
>
>
>
> could you tell me what do you do for this?
>
>
>
> Best Regards,
>
> Evan
>
RE: Client session timed out, have not heard from server in
Posted by yaoxiaohua <ya...@outlook.com>.
Thanks for your reply.
I find spark-env.sh :
SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300 -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash -Dspark.yarn.max.executor.failures=99999 -Dspark.worker.timeout=300"
I just find log like this:
INFO ClientCnxn: Client session timed out, have not heard from server in 40015ms for sessionid 0x351c416297a145a, closing socket connection and attempting reconnect
Before spark2 master process shut down.
I don’t see any zookeeper timeout setting .
Best
From: Yash Sharma [mailto:yash360@gmail.com]
Sent: 2015年12月22日 19:55
To: yaoxiaohua
Cc: user@spark.apache.org
Subject: Re: Client session timed out, have not heard from server in
Hi Evan,
SPARK-9629 referred to connection issues with zookeeper. Could you check if its working fine in your setup.
Also please share other error logs you might be getting.
- Thanks, via mobile, excuse brevity.
On Dec 22, 2015 5:00 PM, "yaoxiaohua" <ya...@outlook.com> wrote:
Hi,
I encounter a similar question, spark1.4
Master2 run some days , then give a timeout exception, then shutdown.
I found a bug :
https://issues.apache.org/jira/browse/SPARK-9629
INFO ClientCnxn: Client session timed out, have not heard from server in 40015ms for sessionid 0x351c416297a145a, closing socket connection and attempting reconnect
could you tell me what do you do for this?
Best Regards,
Evan
Re: Client session timed out, have not heard from server in
Posted by Yash Sharma <ya...@gmail.com>.
Hi Evan,
SPARK-9629 referred to connection issues with zookeeper. Could you check
if its working fine in your setup.
Also please share other error logs you might be getting.
- Thanks, via mobile, excuse brevity.
On Dec 22, 2015 5:00 PM, "yaoxiaohua" <ya...@outlook.com> wrote:
> Hi,
>
> I encounter a similar question, spark1.4
>
> Master2 run some days , then give a timeout exception, then shutdown.
>
> I found a bug :
>
> https://issues.apache.org/jira/browse/SPARK-9629
>
>
>
>
> INFO ClientCnxn: Client session timed out, have not heard from server in
> 40015ms for sessionid 0x351c416297a145a, closing socket connection and
> attempting reconnect
>
>
>
>
>
> could you tell me what do you do for this?
>
>
>
> Best Regards,
>
> Evan
>