You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "HanRyong,Jung (JIRA)" <ji...@apache.org> on 2017/07/26 12:23:00 UTC
[jira] [Updated] (HBASE-18454) Hbase Do not close file descriptor
when using shortcircuit
[ https://issues.apache.org/jira/browse/HBASE-18454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
HanRyong,Jung updated HBASE-18454:
----------------------------------
Description:
I am a user using HDFS 2.7.3, HBASE 1.2.6, centOS 6.8.
The regionserver uses 11 hard disks(jbod) and uses the hbase short circuit.
At this time, when one disk fails in HDFS, I found a phenomenon that I did a hotswap but did not close file descriptor in hbase.
And the fd path on the umount disk is changed to an incorrect path.
If I check /proc/regionserver_pid/fd, if I used /data1/volumn and umounted data1, the path changed to /volumn.
And many file descriptors used in shortcircuit are in the delete state.
example )
ls -al /proc/regionserver_pid/fd
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 946 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490 (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 947 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490_141511919.meta (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 948 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080 (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 949 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080_141513509.meta (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 902 -> /volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir244/subdir160/blk_1257545757 (deleted) <-- change path
when data4 fails, execute fuser)
/sbin/fuser -cu data4
Cannot stat file /proc/regionserver_pid/fd/1275: input/output error
Cannot stat file /proc/regionserver_pid/fd/1282: input/output error
Cannot stat file /proc/regionserver_pid/fd/1283: input/output error
Cannot stat file /proc/regionserver_pid/fd/1297: input/output error
Cannot stat file /proc/regionserver_pid/fd/1298: input/output error
Cannot stat file /proc/regionserver_pid/fd/1306: input/output error
Cannot stat file /proc/regionserver_pid/fd/1335: input/output error
Cannot stat file /proc/regionserver_pid/fd/1336: input/output error
Cannot stat file /proc/regionserver_pid/fd/1343: input/output error
Cannot stat file /proc/regionserver_pid/fd/1344: input/output error
Cannot stat file /proc/regionserver_pid/fd/1381: input/output error
Cannot stat file /proc/regionserver_pid/fd/1382: input/output error
Cannot stat file /proc/regionserver_pid/fd/1383: input/output error
Cannot stat file /proc/regionserver_pid/fd/1384: input/output error
Cannot stat file /proc/regionserver_pid/fd/1394: input/output error
Cannot stat file /proc/regionserver_pid/fd/1395: input/output error
Cannot stat file /proc/regionserver_pid/fd/1410: input/output error
Cannot stat file /proc/regionserver_pid/fd/1430: input/output error
Cannot stat file /proc/regionserver_pid/fd/1477: input/output error
Cannot stat file /proc/regionserver_pid/fd/1672: input/output error
Cannot stat file /proc/regionserver_pid/fd/1673: input/output error
Cannot stat file /proc/regionserver_pid/fd/2006: input/output error
Cannot stat file /proc/regionserver_pid/fd/2021: input/output error
Cannot stat file /proc/regionserver_pid/fd/2083: input/output error
Cannot stat file /proc/regionserver_pid/fd/2309: input/output error
Cannot stat file /proc/regionserver_pid/fd/2416: input/output error
Cannot stat file /proc/regionserver_pid/fd/2428: input/output error
Cannot stat file /proc/regionserver_pid/fd/2540: input/output error
Cannot stat file /proc/regionserver_pid/fd/2555: input/output error
Cannot stat file /proc/regionserver_pid/fd/2621: input/output error
Cannot stat file /proc/regionserver_pid/fd/2624: input/output error
Cannot stat file /proc/regionserver_pid/fd/2756: input/output error
Cannot stat file /proc/regionserver_pid/fd/2803: input/output error
Cannot stat file /proc/regionserver_pid/fd/2927: input/output error
Cannot stat file /proc/regionserver_pid/fd/3456: input/output error
Cannot stat file /proc/regionserver_pid/fd/3534: input/output error
Cannot stat file /proc/regionserver_pid/fd/3814: input/output error
Cannot stat file /proc/regionserver_pid/fd/3891: input/output error
Cannot stat file /proc/regionserver_pid/fd/4034: input/output error
Cannot stat file /proc/regionserver_pid/fd/4337: input/output error
Cannot stat file /proc/regionserver_pid/fd/5272: input/output error
Cannot stat file /proc/regionserver_pid/fd/5864: input/output error
Cannot stat file /proc/regionserver_pid/fd/7419: input/output error
Cannot stat file /proc/regionserver_pid/fd/7422: input/output error
was:
I am a user using HDFS 2.7.3, HBASE 1.2.6, centOS 6.8.
The regionserver uses 11 hard disks(jbod) and uses the hbase short circuit.
At this time, when one disk fails in HDFS, I found a phenomenon that I did a hotswap but did not close file descriptor in hbase.
And the fd path on the umount disk is changed to an incorrect path.
If I check /proc/regionserver_pid/fd, if I used /data1/volumn and umounted data1, the path changed to /volumn.
And many file descriptors used in shortcircuit are in the delete state.
example )
ls -al /proc/regionserver_pid/fd
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 946 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490 (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 947 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490_141511919.meta (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 948 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080 (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 949 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080_141513509.meta (deleted)
lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 902 -> /volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir244/subdir160/blk_1257545757 (deleted)
when data4 fails, execute fuser)
/sbin/fuser -cu data4
Cannot stat file /proc/regionserver_pid/fd/1275: input/output error
Cannot stat file /proc/regionserver_pid/fd/1282: input/output error
Cannot stat file /proc/regionserver_pid/fd/1283: input/output error
Cannot stat file /proc/regionserver_pid/fd/1297: input/output error
Cannot stat file /proc/regionserver_pid/fd/1298: input/output error
Cannot stat file /proc/regionserver_pid/fd/1306: input/output error
Cannot stat file /proc/regionserver_pid/fd/1335: input/output error
Cannot stat file /proc/regionserver_pid/fd/1336: input/output error
Cannot stat file /proc/regionserver_pid/fd/1343: input/output error
Cannot stat file /proc/regionserver_pid/fd/1344: input/output error
Cannot stat file /proc/regionserver_pid/fd/1381: input/output error
Cannot stat file /proc/regionserver_pid/fd/1382: input/output error
Cannot stat file /proc/regionserver_pid/fd/1383: input/output error
Cannot stat file /proc/regionserver_pid/fd/1384: input/output error
Cannot stat file /proc/regionserver_pid/fd/1394: input/output error
Cannot stat file /proc/regionserver_pid/fd/1395: input/output error
Cannot stat file /proc/regionserver_pid/fd/1410: input/output error
Cannot stat file /proc/regionserver_pid/fd/1430: input/output error
Cannot stat file /proc/regionserver_pid/fd/1477: input/output error
Cannot stat file /proc/regionserver_pid/fd/1672: input/output error
Cannot stat file /proc/regionserver_pid/fd/1673: input/output error
Cannot stat file /proc/regionserver_pid/fd/2006: input/output error
Cannot stat file /proc/regionserver_pid/fd/2021: input/output error
Cannot stat file /proc/regionserver_pid/fd/2083: input/output error
Cannot stat file /proc/regionserver_pid/fd/2309: input/output error
Cannot stat file /proc/regionserver_pid/fd/2416: input/output error
Cannot stat file /proc/regionserver_pid/fd/2428: input/output error
Cannot stat file /proc/regionserver_pid/fd/2540: input/output error
Cannot stat file /proc/regionserver_pid/fd/2555: input/output error
Cannot stat file /proc/regionserver_pid/fd/2621: input/output error
Cannot stat file /proc/regionserver_pid/fd/2624: input/output error
Cannot stat file /proc/regionserver_pid/fd/2756: input/output error
Cannot stat file /proc/regionserver_pid/fd/2803: input/output error
Cannot stat file /proc/regionserver_pid/fd/2927: input/output error
Cannot stat file /proc/regionserver_pid/fd/3456: input/output error
Cannot stat file /proc/regionserver_pid/fd/3534: input/output error
Cannot stat file /proc/regionserver_pid/fd/3814: input/output error
Cannot stat file /proc/regionserver_pid/fd/3891: input/output error
Cannot stat file /proc/regionserver_pid/fd/4034: input/output error
Cannot stat file /proc/regionserver_pid/fd/4337: input/output error
Cannot stat file /proc/regionserver_pid/fd/5272: input/output error
Cannot stat file /proc/regionserver_pid/fd/5864: input/output error
Cannot stat file /proc/regionserver_pid/fd/7419: input/output error
Cannot stat file /proc/regionserver_pid/fd/7422: input/output error
> Hbase Do not close file descriptor when using shortcircuit
> ----------------------------------------------------------
>
> Key: HBASE-18454
> URL: https://issues.apache.org/jira/browse/HBASE-18454
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 1.2.6
> Environment: HDFS 2.7.3, HBASE 1.2.6, centOS 6.8
> Reporter: HanRyong,Jung
>
> I am a user using HDFS 2.7.3, HBASE 1.2.6, centOS 6.8.
> The regionserver uses 11 hard disks(jbod) and uses the hbase short circuit.
> At this time, when one disk fails in HDFS, I found a phenomenon that I did a hotswap but did not close file descriptor in hbase.
> And the fd path on the umount disk is changed to an incorrect path.
> If I check /proc/regionserver_pid/fd, if I used /data1/volumn and umounted data1, the path changed to /volumn.
> And many file descriptors used in shortcircuit are in the delete state.
> example )
> ls -al /proc/regionserver_pid/fd
> lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 946 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490 (deleted)
> lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 947 -> /data8/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir21/blk_1215239490_141511919.meta (deleted)
> lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 948 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080 (deleted)
> lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 949 -> /data7/volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir111/subdir27/blk_1215241080_141513509.meta (deleted)
> lr-x------ 1 dragonboy dragonboy 64 2017-07-26 20:54 902 -> /volumn/hdfs/datanode/current/BP-199986352-10.114.243.73-1490077615453/current/finalized/subdir244/subdir160/blk_1257545757 (deleted) <-- change path
> when data4 fails, execute fuser)
> /sbin/fuser -cu data4
> Cannot stat file /proc/regionserver_pid/fd/1275: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1282: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1283: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1297: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1298: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1306: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1335: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1336: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1343: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1344: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1381: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1382: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1383: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1384: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1394: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1395: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1410: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1430: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1477: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1672: input/output error
> Cannot stat file /proc/regionserver_pid/fd/1673: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2006: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2021: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2083: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2309: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2416: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2428: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2540: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2555: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2621: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2624: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2756: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2803: input/output error
> Cannot stat file /proc/regionserver_pid/fd/2927: input/output error
> Cannot stat file /proc/regionserver_pid/fd/3456: input/output error
> Cannot stat file /proc/regionserver_pid/fd/3534: input/output error
> Cannot stat file /proc/regionserver_pid/fd/3814: input/output error
> Cannot stat file /proc/regionserver_pid/fd/3891: input/output error
> Cannot stat file /proc/regionserver_pid/fd/4034: input/output error
> Cannot stat file /proc/regionserver_pid/fd/4337: input/output error
> Cannot stat file /proc/regionserver_pid/fd/5272: input/output error
> Cannot stat file /proc/regionserver_pid/fd/5864: input/output error
> Cannot stat file /proc/regionserver_pid/fd/7419: input/output error
> Cannot stat file /proc/regionserver_pid/fd/7422: input/output error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)