You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2023/02/08 17:59:00 UTC
[jira] [Updated] (HDDS-7930) input stream does not refresh expired block token
[ https://issues.apache.org/jira/browse/HDDS-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-7930:
----------------------------------
Summary: input stream does not refresh expired block token (was: input stream does not update block token)
> input stream does not refresh expired block token
> -------------------------------------------------
>
> Key: HDDS-7930
> URL: https://issues.apache.org/jira/browse/HDDS-7930
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> On a Kerberized cluster HBase ycsb client doesn'twork after the cluster is up for more than 1 day. The RegionServer has the following error which suggests it was due to expired block token used by Ozone input stream.
>
> {noformat}
> 2023-02-08 15:49:51,464 ERROR org.apache.hadoop.hdds.scm.XceiverClientGrpc: Failed to execute command ReadChunk on the pipeline Pipeline[ Id: 39993807-ab46-4996-9313-c397e42dc172, Nodes
> : f244dd2f-678d-4f02-b4c7-021ea7c3a5e5(weichiu-2.weichiu.root.hwx.site/172.27.161.6)f7a94508-4f6e-401f-a50d-8f05570de76a(weichiu-3.weichiu.root.hwx.site/172.27.128.9)8c7429af-315e-49d8-
> 8a64-42ed2b39cb19(weichiu-1.weichiu.root.hwx.site/172.27.74.76), ReplicationConfig: STANDALONE/THREE, State:OPEN, leaderId:8c7429af-315e-49d8-8a64-42ed2b39cb19, CreationTimestamp2023-02
> -07T05:36:46.179Z[UTC]].
> 2023-02-08 15:49:51,465 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 4 locID: 111677748019204244 bcsId: 754837 from pipeline Pip
> elineID=39993807-ab46-4996-9313-c397e42dc172: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
> 2023-02-08 15:49:51,474 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 4 locID: 111677748019204244 bcsId: 754837 from pipeline Pip
> elineID=39993807-ab46-4996-9313-c397e42dc172: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
> 2023-02-08 15:49:51,483 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 1 locID: 111677748019204242 bcsId: 720383 from pipeline Pip
> elineID=ef52af00-6469-4acf-a7ff-b63a98295ba2: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
> 2023-02-08 15:49:51,486 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 2 locID: 111677748019204246 bcsId: 720467 from pipeline Pip
> elineID=ef52af00-6469-4acf-a7ff-b63a98295ba2: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
> 2023-02-08 15:49:51,499 ERROR org.apache.hadoop.hbase.regionserver.StoreFileReader: Bad bloom filter data -- proceeding without
> java.lang.IllegalArgumentException: Failed to load Bloom block
> at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:118)
> at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:94)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:416)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:309)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:242)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:479)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:463)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:245)
> at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1766)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1757)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7053)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:7033)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:3036)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:3016)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2998)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2992)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2624)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2548)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45815)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:384)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:371)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:351)
> Caused by: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:631)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getValidatorList$0(ContainerProtocolCalls.java:638)
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:139)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:291)
> at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:431)
> at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkDataIntoBuffers(ChunkInputStream.java:404)
> at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:389)
> at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:321)
> at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:175)
> at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
> at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:303)
> at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:54)
> at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
> at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
> at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:54)
> at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:64)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:78)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:97)
> at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:233)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1456)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1679)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1490)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1310)
> at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:114)
> ... 22 more {noformat}
> I'm not sure if it's a bug in HBase or Ozone. File as a HDDS bug for now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org