You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nick Dimiduk (JIRA)" <ji...@apache.org> on 2013/09/23 18:06:05 UTC
[jira] [Commented] (HBASE-9639) SecureBulkLoad dispatches file load
requests to all Regions
[ https://issues.apache.org/jira/browse/HBASE-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774670#comment-13774670 ]
Nick Dimiduk commented on HBASE-9639:
-------------------------------------
On the region server, you see the HFile group bulkloaded successfully into one region and then an attempt to load the same file group into others. The source files have been moved into place, to the file not found is thrown for subsequent loads. Which Region loads the files appears to be non-deterministic -- there's a race for whichever region can complete the load first.
{noformat}
2013-08-30 07:37:22,536 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,,1377848151704.ba6689d409333b857c4c06f0b0485ca0.
2013-08-30 07:37:22,655 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Moving hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 to hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
2013-08-30 07:37:22,680 DEBUG [RpcServer.handler=5,port=60020] regionserver.HRegionFileSystem: Committing store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_
2013-08-30 07:37:22,731 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Loaded HFile hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store 'D' as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_ - updating store file list.
2013-08-30 07:37:22,732 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Successfully loaded store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store D (new location: hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_)
2013-08-30 07:37:22,732 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Bulk Load done for: hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
2013-08-30 07:37:22,911 INFO [RpcServer.handler=8,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,x\x00\x00\x00\x00\x00\x00\x00,1377848151709.925bd31a52e5b70a61c2216b5c1d3683.
2013-08-30 07:37:22,914 INFO [RpcServer.handler=4,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,@\x00\x00\x00\x00\x00\x00\x00,1377848151707.57e30d9da6cf9af4763fe9968dfcf564.
org.apache.hadoop.io.MultipleIOException: 3 exceptions [java.io.FileNotFoundException: File does not exist: /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
...
{noformat}
> SecureBulkLoad dispatches file load requests to all Regions
> -----------------------------------------------------------
>
> Key: HBASE-9639
> URL: https://issues.apache.org/jira/browse/HBASE-9639
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
> Affects Versions: 0.95.2
> Environment: Hadoop2, Kerberos
> Reporter: Nick Dimiduk
>
> When running a bulk load on a secure environment and loading data into the first region of a table, the request to load the HFile set is dispatched to all Regions for the table. This is reproduced consistently by running IntegrationTestBulkLoad on a secure cluster. The load fails with an exception that looks like:
> {noformat}
> 2013-08-30 07:37:22,993 INFO [main] mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles, retry attempt 1 with 3 files remaining to group or split
> 2013-08-30 07:37:22,999 ERROR [main] mapreduce.LoadIncrementalHFiles: IOException during splitting
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File does not exist: /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/L/bf41ea13997b4e228d05e67ba7b1b686
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:403)
> at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:284)
> at org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.runLinkedListMRJob(IntegrationTestBulkLoad.java:200)
> at org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.testBulkLoad(IntegrationTestBulkLoad.java:133)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira