You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by marjana <mi...@us.ibm.com> on 2016/09/09 18:06:18 UTC
copyTable mapreduce job failing with ScannerTimeoutException
Hi,
I am trying to copy a table from one cluster to another. This worked fine
for smaller tables, but trying to copy a few of the larger ones, I keep
getting ScannerTimeoutException. It fails at the very end, when it is 98%
complete. Sample from the log:
2016-09-09 17:50:19,065 WARN [Thread-25] mapred.LocalJobRunner:
job_local2143790046_0001
java.lang.Exception: org.apache.hadoop.hbase.client.ScannerTimeoutException:
170435ms passed since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
This is Hbase version 1.2.0.
I did presplit table on the destination cluster.
I tried setting hbase.client.scanner.caching to 1000, but still I get the
timeout exception.
This is not very large table, 400g raw data, some 354068412 rows, only 1
column family.
Anything else I could try?
Thanks
Marjana
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
Looking at the regionserver's log reported as TooBusy, I always see these
periodic Memflushes to free WAL, up to 5 min before the final failure, for
various tables' columns. So I think those must be causing excessive flushes
that regionserver can't keep up with.
I was looking for a parameter to increase # of retries, which is set to 5:
client.AsyncProcess: #18, table=3079_01, attempt=5/4 failed=1ops, last
exception: org.apache.hadoop.hbase.RegionTooBusyException
But don't see any property that would do that.
Thanks
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082702.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by Ted Yu <yu...@gmail.com>.
The log you cited should come from PeriodicMemstoreFlusher.
public static final String MEMSTORE_PERIODIC_FLUSH_INTERVAL =
"hbase.regionserver.optionalcacheflushinterval";
/** Default interval for the memstore flush */
public static final int DEFAULT_CACHE_FLUSH_INTERVAL = 3600000;
Have you find direct correlation between the timestamp of
RegionTooBusyException
and the periodic flush logs ?
Thanks
On Thu, Sep 15, 2016 at 11:39 AM, marjana <mi...@us.ibm.com> wrote:
> So I have tried many different settings on my target cluster. But I still
> get:
> RegionTooBusyException: Above memstore limit
> regionName=3079_01,e4000000,1473944476785.35b4d8395a61af0eefc99f3d7f931c
> d2.,
> server=hb01dn15,16020,1473800758347, memstoreSize=806131464,
> blockingMemStoreSize=805306368
>
> If I look at hb01dn15 for that time, I see a lot of INFO messages,
> "MemStoreFlushercore requesting flush because e has an old edit so flush to
> free WALs after random delay of 89811ms"
> I think these flushes are the cause of regionserver being too busy.
> Our used heap (max 1700m) stays well below maxHeap (6332m).
>
> My nondefault settings:
> <property>
> <name>hbase.replication</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.client.keyvalue.maxsize</name>
> <value>1073741824</value>
> </property>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>180000</value>
> <description>
> </description>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>180000</value>
> <description>
> </description>
> </property>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>6</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>100</value>
> </property>
>
>
> Any suggestions on how to slow down the flushes to free WALs?
> Thanks
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544p4082682.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
So I have tried many different settings on my target cluster. But I still
get:
RegionTooBusyException: Above memstore limit
regionName=3079_01,e4000000,1473944476785.35b4d8395a61af0eefc99f3d7f931cd2.,
server=hb01dn15,16020,1473800758347, memstoreSize=806131464,
blockingMemStoreSize=805306368
If I look at hb01dn15 for that time, I see a lot of INFO messages,
"MemStoreFlushercore requesting flush because e has an old edit so flush to
free WALs after random delay of 89811ms"
I think these flushes are the cause of regionserver being too busy.
Our used heap (max 1700m) stays well below maxHeap (6332m).
My nondefault settings:
<property>
<name>hbase.replication</name>
<value>true</value>
</property>
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>1073741824</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>180000</value>
<description>
</description>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>180000</value>
<description>
</description>
</property>
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>6</value>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>100</value>
</property>
Any suggestions on how to slow down the flushes to free WALs?
Thanks
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082682.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by Ted Yu <yu...@gmail.com>.
You can examine /jmx :
http://search-hadoop.com/m/YGbb3E2a71UVLBK&subj=Re+HBase+Count+Rows+in+Regions+and+Region+Servers
Looks like your load consists of both write and read. Have you turned on
bucket cache ?
http://hbase.apache.org/book.html#offheap.blockcache
On Fri, Sep 9, 2016 at 1:58 PM, marjana <mi...@us.ibm.com> wrote:
> all are defaults:
> <name>hbase.hregion.memstore.block.multiplier</name><value>4</value>
> <name>hbase.hregion.memstore.mslab.enabled</name><value>true</value>
> <name>hbase.hregion.memstore.flush.size</name><value>134217728</value>
>
> Not sure where to see hot spotting (don't have Ambari on this cluster).
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544p4082569.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
all are defaults:
<name>hbase.hregion.memstore.block.multiplier</name><value>4</value>
<name>hbase.hregion.memstore.mslab.enabled</name><value>true</value>
<name>hbase.hregion.memstore.flush.size</name><value>134217728</value>
Not sure where to see hot spotting (don't have Ambari on this cluster).
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082569.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by Ted Yu <yu...@gmail.com>.
The 'Above memstore limit' warning meant that your region server(s) was
under pressure of the write load.
Can you share memstore related config parameters ?
Did you observe hot spotting in the region server(s) ?
Cheers
On Fri, Sep 9, 2016 at 1:20 PM, marjana <mi...@us.ibm.com> wrote:
> Looked at regionserver logs and found different errors:
>
> 2016-09-09 14:52:26,172 WARN [hconnection-0x41059e91-
> shared--pool3-t500214]
> client.AsyncProcess: #2, table=3079_01, attempt=5/4 failed=1ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=3079_01,aeeeeee4,1473428857901.3042d01acd1bd7066934a3374ce1a7
> 4a.,
> server=hb01dn25,16020,1471966355932, memstoreSize=537147864,
> blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:
> 3630)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2858)
> .....
> Fri Sep 09 14:52:24 UTC 2016; not retrying 1 - final failure
> 2016-09-09 14:52:26,172 ERROR
> [ReplicationRpcServer.handler=0,queue=0,port=16020]
> regionserver.ReplicationSink: Unable to accept edit because:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed
> 1 action: RegionTooBusyException: 1 time,
>
>
> My CopyTable ended some 3 hours later (with the error it failed). I wonder
> why it hasn't errored out at 14:52.
> Will try setting hbase.regionserver.lease.period and hbase.rpc.timeout to
> double the default, 120 seconds on the target cluster.
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544p4082563.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
Looked at regionserver logs and found different errors:
2016-09-09 14:52:26,172 WARN [hconnection-0x41059e91-shared--pool3-t500214]
client.AsyncProcess: #2, table=3079_01, attempt=5/4 failed=1ops, last
exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
regionName=3079_01,aeeeeee4,1473428857901.3042d01acd1bd7066934a3374ce1a74a.,
server=hb01dn25,16020,1471966355932, memstoreSize=537147864,
blockingMemStoreSize=536870912
at
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3630)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2858)
.....
Fri Sep 09 14:52:24 UTC 2016; not retrying 1 - final failure
2016-09-09 14:52:26,172 ERROR
[ReplicationRpcServer.handler=0,queue=0,port=16020]
regionserver.ReplicationSink: Unable to accept edit because:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: RegionTooBusyException: 1 time,
My CopyTable ended some 3 hours later (with the error it failed). I wonder
why it hasn't errored out at 14:52.
Will try setting hbase.regionserver.lease.period and hbase.rpc.timeout to
double the default, 120 seconds on the target cluster.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082563.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
It runs for 4 hours. There's nothing useful in the mapreduce log, at least
nothing I see.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082551.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by Ted Yu <yu...@gmail.com>.
How long was your mapreduce job ?
You may need to check log for map tasks to get more information.
Thanks
On Fri, Sep 9, 2016 at 11:17 AM, marjana <mi...@us.ibm.com> wrote:
> I haven't tried that, afraid of how it will affect client connections.
> Any idea why it fails at the very end?
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544p4082548.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by marjana <mi...@us.ibm.com>.
I haven't tried that, afraid of how it will affect client connections.
Any idea why it fails at the very end?
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-tp4082544p4082548.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: copyTable mapreduce job failing with ScannerTimeoutException
Posted by Ted Yu <yu...@gmail.com>.
Have you tried increasing the value of hbase.client.scanner.timeout.period
(default 60000) ?
On Fri, Sep 9, 2016 at 11:06 AM, marjana <mi...@us.ibm.com> wrote:
> Hi,
> I am trying to copy a table from one cluster to another. This worked fine
> for smaller tables, but trying to copy a few of the larger ones, I keep
> getting ScannerTimeoutException. It fails at the very end, when it is 98%
> complete. Sample from the log:
> 2016-09-09 17:50:19,065 WARN [Thread-25] mapred.LocalJobRunner:
> job_local2143790046_0001
> java.lang.Exception: org.apache.hadoop.hbase.client.
> ScannerTimeoutException:
> 170435ms passed since the last invocation, timeout is currently set to
> 60000
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(
> LocalJobRunner.java:462)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> This is Hbase version 1.2.0.
> I did presplit table on the destination cluster.
> I tried setting hbase.client.scanner.caching to 1000, but still I get the
> timeout exception.
> This is not very large table, 400g raw data, some 354068412 rows, only 1
> column family.
> Anything else I could try?
> Thanks
> Marjana
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544.html
> Sent from the HBase User mailing list archive at Nabble.com.
>