You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2017/10/17 20:00:01 UTC
[jira] [Resolved] (HBASE-19025) Hbase MasterProcWALs getting pile
up
[ https://issues.apache.org/jira/browse/HBASE-19025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Elser resolved HBASE-19025.
--------------------------------
Resolution: Invalid
Please leave JIRA issues for making code changes to HBase, and take questions like these to the mailing lists.
Looks like you need to increase the number of open files allowed via ulimit on your datanodes.
> Hbase MasterProcWALs getting pile up
> ------------------------------------
>
> Key: HBASE-19025
> URL: https://issues.apache.org/jira/browse/HBASE-19025
> Project: HBase
> Issue Type: Bug
> Components: hbase
> Affects Versions: 1.2.6
> Reporter: Irfan
>
> MasterProcWALs getting pile up and getting below error in Hmaster log
> 2017-10-17 03:34:42,141 INFO [ip-176-0-0-44:16000.activeMasterManager] hdfs.DFSClient: Successfully connected to /176.0.0.29:50010 for BP-1665161556-176.0.
> 0.44-1498805562805:blk_1073773522_32746
> 2017-10-17 03:34:42,182 INFO [ip-176-0-0-44:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://192.168.168.1:9000/hbase2/MasterProcWALs/state-00000000000000005592.log
> 2017-10-17 03:34:42,182 INFO [ip-176-0-0-44:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://ec2-34-195-113-113.compu
> te-1.amazonaws.com:9000/hbase2/MasterProcWALs/state-00000000000000005592.log after 0ms
> 2017-10-17 03:34:42,184 WARN [ip-176-0-0-44:16000.activeMasterManager] hdfs.BlockReaderFactory: I/O error constructing remote block reader.
> java.io.IOException: Got error, status message opReadBlock BP-1665161556-176.0.0.44-1498805562805:blk_1073773523_32747 received exception java.io.FileNotFou
> ndException: /data/datanode/data/current/BP-1665161556-176.0.0.44-1498805562805/current/finalized/subdir0/subdir123/blk_1073773523_32747.meta (Too many open
> files), for OP_READ_BLOCK, self=/176.0.0.44:59340, remote=/176.0.0.44:50010, for file /hbase2/MasterProcWALs/state-00000000000000005592.log, for pool BP-16
> 65161556-176.0.0.44-1498805562805 block 1073773523_32747
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)