You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vlad Sharanhovich (JIRA)" <ji...@apache.org> on 2015/07/06 06:22:05 UTC

[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614523#comment-14614523 ] 

Vlad Sharanhovich commented on MAPREDUCE-6363:
----------------------------------------------

This fix is not resolving the problem but rather alleviating it using unique output file names for each mapper to avoid write conflicts when multiple mappers are running on the name node. Now, nor usage of random number (OMG!), not TaskID would generate a stable output file names that can be (re-)used for following nnbench runs of READ and DELETE operations. This means that only CREATE will work, everything else is still broken.

> [NNBench] Lease mismatch error when running with multiple mappers
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-6363
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: benchmarks
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>            Priority: Critical
>             Fix For: 2.8.0
>
>         Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, MAPREDUCE-6363-002.patch
>
>
> Command :
> ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 -bytesToWrite 1024000000 -baseDir /benchmarks/NNBench`hostname -s` -replicationFactorPerFile 3 -maps 100 -reduces 10
> Trace :
> 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.105.214:36320: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)