HDFS - block key missing

[Robert Mroczkowski <ro...@allegrogroup.com>]

Hi,

After restart of one datanode I noticed problems with some writes.
DN logs:

2014-04-09 09:57:31,090 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1556031262-10.242.1.238-1363618396299:blk_-9057573077887108744_445939099 src: /10.68.58.16:60963 dest: /10.68.58.10:1004
2014-04-09 09:57:31,090 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification failed: op=WRITE_BLOCK, remoteAddress=/10.68.58.16:60963, message=Can't re-compute password for block_token_identifier (expiryDate=1397066251086, keyId=1668559453, userId=hdfs, blockPoolId=BP-1556031262-10.242.1.238-1363618396299, blockId=-9057573077887108744, access modes=[WRITE]), since the required block key (keyID=1668559453) doesn't exist.
014-04-09 09:57:31,090 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: s40821.dc4.local:1004:DataXceiver error processing WRITE_BLOCK operation  src: /10.68.58.16:60963 dest: /10.68.58.10:1004
org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't re-compute password for block_token_identifier (expiryDate=1397066251086, keyId=1668559453, userId=hdfs, blockPoolId=BP-1556031262-10.242.1.238-1363618396299, blockId=-9057573077887108744, access modes=[WRITE]), since the required block key (keyID=1668559453) doesn't exist.
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:302)
        at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:97)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:952)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:443)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:103)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:67)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
        at java.lang.Thread.run(Thread.java:724)

How can we fix it ?

Best wishes,
   Robert



---
Robert Mroczkowski
System Administrator
Centrum IT Allegro w Toruniu
http://www.allegro.pl/

Grupa Allegro Sp. z o.o. z siedzibą w Poznaniu, 60-324 Poznań, przy ul.
Marcelińskiej 90, wpisana do rejestru przedsiębiorców prowadzonego przez
Sąd Rejonowy Poznań - Nowe Miasto i Wilda, Wydział VIII Gospodarczy
Krajowego Rejestru Sądowego pod numerem KRS 0000268796, o kapitale
zakładowym w wysokości 33 474 500 zł, posiadająca numer identyfikacji
podatkowej NIP: 5272525995.