Streaming: LeaseExpiredException when writing checkpoint

[Lin Zhao <li...@exabeam.com>]

I'm seeing this error in the driver when running a streaming job. Not sure If it's critical.

It happens maybe half of time checkpoint is saved. There are retries in the log but sometimes results in "Could not write checkpoint for time 1454006320000 ms to file hdfs://ip-172-31-35-122.us-west-2.compute.internal:8020/user/exabeam/checkpoint-1454006320000". Any help in understanding this error is appreciated.


16/01/28 18:38:40 INFO CheckpointWriter: Saving checkpoint for time 1454006320000 ms to file 'hdfs://ip-172-31-35-122.us-west-2.compute.internal:8020/user/exabeam/checkpoint-1454006320000'

16/01/28 18:38:40 WARN DFSClient: DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/exabeam/temp (inode 2058161): File does not exist. [Lease.  Holder: DFSClient_NONMAPR

EDUCE_762594086_1, pendingcreates: 2]

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3605)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3402)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3258)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:668)

        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)


        at org.apache.hadoop.ipc.Client.call(Client.java:1468)

        at org.apache.hadoop.ipc.Client.call(Client.java:1399)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

        at com.sun.proxy.$Proxy19.addBlock(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)

        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy20.addBlock(Unknown Source)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)