You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2018/05/14 02:56:00 UTC
[jira] [Created] (IMPALA-7021) Impala query fails with
BlockMissingException while HDFS throws SSLHandshakeException
Mostafa Mokhtar created IMPALA-7021:
---------------------------------------
Summary: Impala query fails with BlockMissingException while HDFS throws SSLHandshakeException
Key: IMPALA-7021
URL: https://issues.apache.org/jira/browse/IMPALA-7021
Project: IMPALA
Issue Type: Bug
Components: Backend, Security
Affects Versions: Impala 2.12.0
Reporter: Mostafa Mokhtar
Assignee: Sailesh Mukil
Impala query failed with status
{code}
Status: Disk I/O error: Error reading from HDFS file: hdfs://vc1512.halxg.cloudera.com:8020/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq Error(255): Unknown error 255 Root cause: BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803316_11063749 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq
{code}
Impala log
{code}
I0513 19:50:07.248292 150900 coordinator.cc:1092] Release admission control resources for query_id=f14b2814eea7b58b:c2eeddb400000000
W0513 19:50:11.313998 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
Java exception follows:
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
W0513 19:50:11.317173 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
Java exception follows:
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
W0513 19:50:11.318079 109914 DFSInputStream.java:1275] Connection failure: Failed to connect to /10.17.187.36:20002 for file /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq for block BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772:org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
{code}
Impala error log
{code}
hdfsPread: FSDataInputStream#read error:
BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parqorg.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1048)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1031)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1010)
at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1156)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
{code}
NN log
{code}
2018-05-13 19:51:54,857 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 11 millisecond(s).
2018-05-13 19:51:57,409 WARN org.mortbay.log: EXCEPTION
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2018-05-13 19:52:00,591 INFO BlockStateChange: BLOCK* processReport 0x422a616a86ad5ba4: discarded non-initial block report from DatanodeRegistration(10.17.187.39, datanodeUuid=42d8151a-7aaf-4f78-90d8-7d1bda177d5e, infoPort=0, infoSecurePort=20004, ipcPort=20001, storageInfo=lv=-56;cid=cluster2;nsid=606873798;c=0) because namenode still in startup phase
{code}
Datanode log
{code}
2018-05-13 19:50:08,946 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing un
known operation src: /10.17.246.46:44103 dst: /10.17.187.40:20002
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncr
yptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=1802676210) doesn't exist. Current key: 180
2676198]
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:372)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the re
quired block key (keyID=1802676210) doesn't exist. Current key: 1802676198
at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrieveDataEncryptionKey(BlockTokenSecretManager.java:430)
at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrieveDataEncryptionKey(BlockPoolTokenSecretManager.java:130)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptionKeyFromUserName(SaslDataTransferServer.java:267)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$000(SaslDataTransferServer.java:71)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$1.apply(SaslDataTransferServer.java:175)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:239)
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 7 more
2018-05-13 19:50:08,949 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing unknown operation src: /10.17.246.46:44104 dst: /10.17.187.40:20002
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)