You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Manjunath Anand (JIRA)" <ji...@apache.org> on 2017/01/28 15:11:24 UTC
[jira] [Commented] (HDFS-11367) AlreadyBeingCreatedException
"current leaseholder is trying to recreate file" when trying to append to
file
[ https://issues.apache.org/jira/browse/HDFS-11367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844089#comment-15844089 ]
Manjunath Anand commented on HDFS-11367:
----------------------------------------
HDFS supports single writer per file at any given point of time as can be seen in the JIRA https://issues.apache.org/jira/browse/HDFS-8177.
Even if you use a single thread per file and if that thread creates a file and using the returned OutputStream writes data to file then the file will be having an under construction feature and a lease at the time of creating the file which will be cached. If the same thread tries to do an append without closing the OutputStream then the HDFS code finds that the file is underconstruction with a valid lease and doesnot another lease to be created by the same thread.
Once you get the OutputStream object from your getOutputStream method and once done writing data, close the OutputStream . This will make sure the underconstruction feature of the file and lease is removed. Then calling the getOutputStream later will result in append and would not result in this kind of error.
> AlreadyBeingCreatedException "current leaseholder is trying to recreate file" when trying to append to file
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-11367
> URL: https://issues.apache.org/jira/browse/HDFS-11367
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.5.0
> Environment: Red Hat Enterprise Linux Server release 6.8
> Reporter: Dmitry Goldenberg
> Assignee: Manjunath Anand
>
> We have code which creates a file in HDFS and continuously appends lines to the file, then closes the file at the end. This is done by a single dedicated thread.
> We specifically instrumented the code to make sure only one 'client'/thread ever writes to the file because we were seeing "current leaseholder is trying to recreate file" errors.
> For some background see this for example: https://community.cloudera.com/t5/Storage-Random-Access-HDFS/How-to-append-files-to-HDFS-with-Java-quot-current-leaseholder/m-p/41369
> This issue is very critical to us as any error terminates a mission critical application in production.
> Intermittently, we see the below exception, regardless of what our code is doing which is create the file, keep appending, then close:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): failed to create file /data/records_20170125_1.txt for DFSClient_NONMAPREDUCE_-167421175_1 for client 1XX.2XX.1XX.XXX because current leaseholder is trying to recreate file.
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3075)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2905)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3189)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3153)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:612)
> at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:125)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:414)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy24.append(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy24.append(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:282)
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1586)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1626)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1614)
> at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:313)
> at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:309)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:309)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
> at com.myco.MyAppender.getOutputStream(MyAppender.java:147)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org