You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by wa...@intec.co.jp on 2017/04/07 04:15:39 UTC
Is it because the JVM memory is low?
Hello.
I am testing HBASE now.
The following problem occurred.
I do not know why this problem occurred.
Is it because the JVM memory is low?
(The memory size of HBASE is the default(1GB).)
Please give me on solution.
(Problem)
When registering data of a 700MiB file, subsequent data registration fails.
(Pseudo-Distributed environment)
HBase : 1.2.4
Hadoop : 2.7.3
java version : 1.8.0_121
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
(Server)
OS:Red Hat Enterprise Linux Server release 6.7 (Santiago)
\u3000\uff08Linux patdevsrv02 2.6.32-573.el6.x86_64\uff09
CPU:Intel(R) Xeon(R) CPU E5640 @2.67GHz * 2
Memory:12GB
Swap:14GB
Network:1Gbps
(Log)
(1) java_program.log
2017-04-07 11:33:12,056 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=10, retries=35, started=38618 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:33:22,152 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=11, retries=35, started=48716 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:34:00,532 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t2] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=10, retries=35, started=38270 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:34:10,547 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t2] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=11, retries=35, started=48284 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:34:49,088 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t3] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=10, retries=35, started=38337 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:34:59,143 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t3] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=11, retries=35, started=48392 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:35:37,840 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t4] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=10, retries=35, started=38256 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
2017-04-07 11:35:47,912 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t4] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception, tries=11, retries=35, started=48328 ms ago, cancelled=false, msg=row 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385, seqNum=0
:
:
(2) hbase-test-master-example.log
2017-04-07 11:34:38,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053376 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:39,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053377 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:40,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053378 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:41,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053379 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:42,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053380 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:43,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/RESCAN0000053381 entered state: DONE example,16000,1491532078399
2017-04-07 11:34:44,082 INFO [example,16000,1491532078399_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 4 unassigned = 4 tasks={/hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%2Fexample%252C16201%252C1491532079385..meta.1491532095844.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0, /hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%2Fexample%252C16201%252C1491532079385..meta.1491532101210.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0, /hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%2Fexample%252C16201%252C1491532079385..meta.1491532100787.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%2Fexample%252C16201%252C1491532079385..meta.1491532101137.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0}
:
:
Thank you.
Tsutomu Wakuda
Re: Is it because the JVM memory is low?
Posted by Ted Yu <yu...@gmail.com>.
Hi, Tsutomu:
Do you keep memstore parameters as default ?
Can you pastebin the remaining log from master ?
See if you can give master more than 1GB of memory.
Cheers
On Thu, Apr 6, 2017 at 9:15 PM, <wa...@intec.co.jp> wrote:
> Hello.
>
> I am testing HBASE now.
>
> The following problem occurred.
>
> I do not know why this problem occurred.
>
> Is it because the JVM memory is low?
> (The memory size of HBASE is the default(1GB).)
>
> Please give me on solution.
>
> (Problem)
>
> When registering data of a 700MiB file, subsequent data registration fails.
>
> (Pseudo-Distributed environment)
> HBase : 1.2.4
> Hadoop : 2.7.3
> java version : 1.8.0_121
> Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
>
> (Server)
> OS:Red Hat Enterprise Linux Server release 6.7 (Santiago)
> （Linux patdevsrv02 2.6.32-573.el6.x86_64）
> CPU:Intel(R) Xeon(R) CPU E5640 @2.67GHz * 2
> Memory:12GB
> Swap:14GB
> Network:1Gbps
>
> (Log)
> (1) java_program.log
>
> 2017-04-07 11:33:12,056 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t1]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=10, retries=35, started=38618 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:33:22,152 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t1]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=11, retries=35, started=48716 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:34:00,532 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t2]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=10, retries=35, started=38270 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:34:10,547 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t2]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=11, retries=35, started=48284 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:34:49,088 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t3]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=10, retries=35, started=38337 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:34:59,143 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t3]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=11, retries=35, started=48392 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:35:37,840 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t4]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=10, retries=35, started=38256 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> 2017-04-07 11:35:47,912 [hconnection-0x4c05f83b-metaLookup-shared--pool2-t4]
> INFO org.apache.hadoop.hbase.client.RpcRetryingCaller : Call exception,
> tries=11, retries=35, started=48328 ms ago, cancelled=false, msg=row
> 'test_table,test_key_1,99999999999999' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=example,16201,1491532079385,
> seqNum=0
> :
> :
>
> (2) hbase-test-master-example.log
>
> 2017-04-07 11:34:38,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053376 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:39,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053377 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:40,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053378 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:41,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053379 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:42,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053380 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:43,084 INFO [main-EventThread] coordination.SplitLogManagerCoordination:
> task /hbase/splitWAL/RESCAN0000053381 entered state: DONE
> example,16000,1491532078399
> 2017-04-07 11:34:44,082 INFO [example,16000,1491532078399_
> splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 4
> unassigned = 4 tasks={/hbase/splitWAL/WALs%2Fexample%2C16201%
> 2C1491532079385-splitting%2Fexample%252C16201%252C1491532079385..meta.1491532095844.meta=last_update
> = -1 last_version = -1 cur_worker_name = null status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%
> 2Fexample%252C16201%252C1491532079385..meta.1491532101210.meta=last_update
> = -1 last_version = -1 cur_worker_name = null status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%
> 2Fexample%252C16201%252C1491532079385..meta.1491532100787.meta=last_update
> = -1 last_version = -1 cur_worker_name = null status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fexample%2C16201%2C1491532079385-splitting%
> 2Fexample%252C16201%252C1491532079385..meta.1491532101137.meta=last_update
> = -1 last_version = -1 cur_worker_name = null status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 4 done = 0 error = 0}
> :
> :
>
> Thank you.
>
> Tsutomu Wakuda
>
>