You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/12/23 21:46:00 UTC
[jira] [Resolved] (IMPALA-7021) Impala query fails with
BlockMissingException while HDFS throws SSLHandshakeException
[ https://issues.apache.org/jira/browse/IMPALA-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-7021.
-----------------------------------
Resolution: Cannot Reproduce
> Impala query fails with BlockMissingException while HDFS throws SSLHandshakeException
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-7021
> URL: https://issues.apache.org/jira/browse/IMPALA-7021
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Security
> Affects Versions: Impala 2.12.0
> Reporter: Mostafa Mokhtar
> Priority: Major
>
> Impala query failed with status
> {code}
> Status: Disk I/O error: Error reading from HDFS file: hdfs://vc1512.halxg.cloudera.com:8020/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq Error(255): Unknown error 255 Root cause: BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803316_11063749 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq
> {code}
> Impala log
> {code}
> I0513 19:50:07.248292 150900 coordinator.cc:1092] Release admission control resources for query_id=f14b2814eea7b58b:c2eeddb400000000
> W0513 19:50:11.313998 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
> Java exception follows:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
> at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
> at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
> at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
> at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
> at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
> at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> W0513 19:50:11.317173 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
> Java exception follows:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
> at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
> at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
> at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
> at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
> at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
> at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> W0513 19:50:11.318079 109914 DFSInputStream.java:1275] Connection failure: Failed to connect to /10.17.187.36:20002 for file /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq for block BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772:org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
> {code}
> Impala error log
> {code}
> hdfsPread: FSDataInputStream#read error:
> BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parqorg.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
> at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1048)
> at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1031)
> at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1010)
> at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1156)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> {code}
> NN log
> {code}
> 2018-05-13 19:51:54,857 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 11 millisecond(s).
> 2018-05-13 19:51:57,409 WARN org.mortbay.log: EXCEPTION
> javax.net.ssl.SSLHandshakeException: no cipher suites in common
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
> at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045)
> at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741)
> at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 2018-05-13 19:52:00,591 INFO BlockStateChange: BLOCK* processReport 0x422a616a86ad5ba4: discarded non-initial block report from DatanodeRegistration(10.17.187.39, datanodeUuid=42d8151a-7aaf-4f78-90d8-7d1bda177d5e, infoPort=0, infoSecurePort=20004, ipcPort=20001, storageInfo=lv=-56;cid=cluster2;nsid=606873798;c=0) because namenode still in startup phase
> {code}
> Datanode log
> {code}
> 2018-05-13 19:50:08,946 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing un
> known operation src: /10.17.246.46:44103 dst: /10.17.187.40:20002
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncr
> yptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=1802676210) doesn't exist. Current key: 180
> 2676198]
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:372)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the re
> quired block key (keyID=1802676210) doesn't exist. Current key: 1802676198
> at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrieveDataEncryptionKey(BlockTokenSecretManager.java:430)
> at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrieveDataEncryptionKey(BlockPoolTokenSecretManager.java:130)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptionKeyFromUserName(SaslDataTransferServer.java:267)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$000(SaslDataTransferServer.java:71)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$1.apply(SaslDataTransferServer.java:175)
> at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:239)
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 7 more
> 2018-05-13 19:50:08,949 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing unknown operation src: /10.17.246.46:44104 dst: /10.17.187.40:20002
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)