You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Motty Cruz <mo...@gmail.com> on 2017/03/02 18:02:47 UTC
last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
Hello, in the past two weeks, I see the following error on HBase Thrift
servers, we have total of about 10 Thrift servers and randomly get the
following errors:
2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess:
#4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on
local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on
e26-node.fqdn.com,60020,1487800633128, tracking started null,
retrying after=10060ms, replay=2ops
2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess:
#4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
gress=22
Restarting the Thrift server resolves the issue. Any ideas where should I be
looking for?
Thanks,
Motty
Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020
failed on local exception
Posted by Ted Yu <yu...@gmail.com>.
How many Thrift servers did you start ?
The log snippet was INFO logs which didn't give us more clue.
Can you pastebin more of the logs (from Thrift servers) ?
Thanks
On Wed, Mar 8, 2017 at 11:20 AM, Motty Cruz <mo...@gmail.com> wrote:
> Hello, in the past three weeks I have seen the following errors on HBase,
> the Thrift servers stop talking to each other. Anybody seen this?
>
>
>
> Mar 8, 5:31:49.660 AM INFO org.apache.hadoop.hbase.
> client.AsyncProcess
>
> #7147032, waiting for some tasks to finish. Expected max=0,
> tasksInProgress=25
>
> Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase.
> client.AsyncProcess
>
> #7143730, table=LogSum, attempt=32/35 succeeded on h24-node.fqdn.com
> ,60020,1474305457742, tracking started Wed Mar 08 03:52:39 PST 2017
>
> Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase.
> client.AsyncProcess
>
> #7143730, waiting for some tasks to finish. Expected max=0,
> tasksInProgress=21
>
>
>
> Any ideas?
>
>
>
> Thanks,
> Motty
>
>
>
>
>
> From: Ravi Prakash [mailto:ravihadoop@gmail.com]
> Sent: Tuesday, March 07, 2017 11:32 AM
> To: Motty Cruz
> Cc: user
> Subject: Re: last exception: java.io.IOException: Call to
> e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
>
>
>
> You should probably email Hbase mailing lists rather than Hadoop :
> https://hbase.apache.org/mail-lists.html
>
>
>
> On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo...@gmail.com> wrote:
>
> Hello, in the past two weeks, I see the following error on HBase Thrift
> servers, we have total of about 10 Thrift servers and randomly get the
> following errors:
>
>
>
> 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
>
> va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on
> local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
>
> on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on
> e26-node.fqdn.com,60020,1487800633128, tracking started null,
>
> retrying after=10060ms, replay=2ops
>
> 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
>
> gress=22
>
>
>
> Restarting the Thrift server resolves the issue. Any ideas where should I
> be looking for?
>
>
>
> Thanks,
> Motty
>
>
>
>
RE: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
Posted by Motty Cruz <mo...@gmail.com>.
Hello, in the past three weeks I have seen the following errors on HBase, the Thrift servers stop talking to each other. Anybody seen this?
Mar 8, 5:31:49.660 AM INFO org.apache.hadoop.hbase.client.AsyncProcess
#7147032, waiting for some tasks to finish. Expected max=0, tasksInProgress=25
Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase.client.AsyncProcess
#7143730, table=LogSum, attempt=32/35 succeeded on h24-node.fqdn.com,60020,1474305457742, tracking started Wed Mar 08 03:52:39 PST 2017
Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase.client.AsyncProcess
#7143730, waiting for some tasks to finish. Expected max=0, tasksInProgress=21
Any ideas?
Thanks,
Motty
From: Ravi Prakash [mailto:ravihadoop@gmail.com]
Sent: Tuesday, March 07, 2017 11:32 AM
To: Motty Cruz
Cc: user
Subject: Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
You should probably email Hbase mailing lists rather than Hadoop : https://hbase.apache.org/mail-lists.html
On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo...@gmail.com> wrote:
Hello, in the past two weeks, I see the following error on HBase Thrift servers, we have total of about 10 Thrift servers and randomly get the following errors:
2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on e26-node.fqdn.com,60020,1487800633128, tracking started null,
retrying after=10060ms, replay=2ops
2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
gress=22
Restarting the Thrift server resolves the issue. Any ideas where should I be looking for?
Thanks,
Motty
Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020
failed on local exception
Posted by Ravi Prakash <ra...@gmail.com>.
You should probably email Hbase mailing lists rather than Hadoop :
https://hbase.apache.org/mail-lists.html
On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo...@gmail.com> wrote:
> Hello, in the past two weeks, I see the following error on HBase Thrift
> servers, we have total of about 10 Thrift servers and randomly get the
> following errors:
>
>
>
> 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
>
> va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on
> local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
>
> on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on
> e26-node.fqdn.com,60020,1487800633128, tracking started null,
>
> retrying after=10060ms, replay=2ops
>
> 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
>
> gress=22
>
>
>
> Restarting the Thrift server resolves the issue. Any ideas where should I
> be looking for?
>
>
>
> Thanks,
> Motty
>
RE: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
Posted by Motty Cruz <mo...@gmail.com>.
hbase version
HBase 1.2.0-cdh5.7.2
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Thursday, March 02, 2017 10:07 AM
To: Motty Cruz; user@hbase.apache.org
Subject: Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
Dropping hadoop ML.
This should be directed to user@hbase.
Which hbase release are you using ?
On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo...@gmail.com> wrote:
Hello, in the past two weeks, I see the following error on HBase Thrift servers, we have total of about 10 Thrift servers and randomly get the following errors:
2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on e26-node.fqdn.com,60020,1487800633128, tracking started null,
retrying after=10060ms, replay=2ops
2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
gress=22
Restarting the Thrift server resolves the issue. Any ideas where should I be looking for?
Thanks,
Motty
Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020
failed on local exception
Posted by Ted Yu <yu...@gmail.com>.
Dropping hadoop ML.
This should be directed to user@hbase.
Which hbase release are you using ?
On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo...@gmail.com> wrote:
> Hello, in the past two weeks, I see the following error on HBase Thrift
> servers, we have total of about 10 Thrift servers and randomly get the
> following errors:
>
>
>
> 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja
>
> va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on
> local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti
>
> on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on
> e26-node.fqdn.com,60020,1487800633128, tracking started null,
>
> retrying after=10060ms, replay=2ops
>
> 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess:
> #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro
>
> gress=22
>
>
>
> Restarting the Thrift server resolves the issue. Any ideas where should I
> be looking for?
>
>
>
> Thanks,
> Motty
>