You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Yang <te...@gmail.com> on 2014/10/07 23:55:06 UTC
SSVD: lease conflict due to 2 attempts using the same dir
we are running mahout ssvd, with a --tempDir parameter,
but we are constantly getting this error below.
the following seems to suggest that the 2 attempts in mapper are trying to
access the same tmp dir on hdfs.
if this is the cause, then ssvd has a bug in that different attempts should
use different temp dirs . right?
I'm trying now without the --tempDir param
thanks
Yang
Error:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
Lease mismatch on /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1 but is
accessed by DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1 at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
org.apache.hadoop.ipc.Client.call(Client.java:1410) at
org.apache.hadoop.ipc.Client.call(Client.java:1363) at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy15.complete(Unknown Source) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy15.complete(Unknown Source) at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
at
org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
at
org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
at
org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Looks like pecularities of a newer hadoop stack.. never saw this before.
On Oct 7, 2014 2:55 PM, "Yang" <te...@gmail.com> wrote:
> we are running mahout ssvd, with a --tempDir parameter,
> but we are constantly getting this error below.
>
> the following seems to suggest that the 2 attempts in mapper are trying to
> access the same tmp dir on hdfs.
>
> if this is the cause, then ssvd has a bug in that different attempts should
> use different temp dirs . right?
>
>
> I'm trying now without the --tempDir param
>
> thanks
> Yang
>
> Error:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> Lease mismatch on /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1 but is
> accessed by DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1 at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
>
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> org.apache.hadoop.ipc.Client.call(Client.java:1363) at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> at
>
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> at
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> at
>
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> at
>
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> at
>
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> at
>
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
>
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
RE: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
This is using side input, yes. But it is standard practice. For example,
map side joins do basically the same.
Specifically wrt opportunistic execution this should be fine. Hdfs does not
disallow opening the same file for reading by multiple tasks iirc.
Sent from my phone.
On Oct 9, 2014 1:02 PM, "Ken Krugler" <kk...@transpac.com> wrote:
> The BtJob's BtMapper has some "interesting" logic in its setup routine,
> where it looks like it's creating a side-channel:
>
> /*
> * actually this is kind of dangerous because this routine thinks we
> need
> * to create file name for our current job and this will use -m- so
> it's
> * just serendipity we are calling it from the mapper too as the
> QJob did.
> */
> Path qInputPath =
> new Path(qJobPath, FileOutputFormat.getUniqueFile(context,
> QJob.OUTPUT_QHAT,
> ""));
> blockNum = context.getTaskAttemptID().getTaskID().getId();
>
> SequenceFileValueIterator<DenseBlockWritable> qhatInput =
> new SequenceFileValueIterator<DenseBlockWritable>(qInputPath,
> true,
> conf);
> closeables.addFirst(qhatInput);
>
> It wouldn't surprise me if this fails when speculative execution is turned
> on.
>
> -- Ken
>
>
> > From: Dmitriy Lyubimov
> > Sent: October 9, 2014 11:31:42am PDT
> > To: user@mahout.apache.org
> > Subject: Re: SSVD: lease conflict due to 2 attempts using the same dir
> >
> > wow. good to know. however, it would seem to me like a bug in
> > MultipleOutputs? either way, it doesn't seem anything to do with the
> Mahout
> > code itself.
> >
> > On Thu, Oct 9, 2014 at 10:32 AM, Yang <te...@gmail.com> wrote:
> >
> >> yes, that's what I'm saying, I disabled speculative, it works for now
> (kind
> >> of is a hack)
> >>
> >>
> >> also yes, this is hadoop 2.0 with YARN
> >>
> >> this has nothing to do with overwrite mode. the 2 attempts are run
> >> simultaneously because they are speculative runs
> >>
> >> On Wed, Oct 8, 2014 at 12:07 AM, Serega Sheypak <
> serega.sheypak@gmail.com>
> >> wrote:
> >>
> >>> owned by
> >>> DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1
> >>> *but is
> >>> accessed by
> >>> DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1*
> >>> at
> >>>
> >>> Looks like two clients try to write to the same dir and they are trying
> >> to
> >>> write to the same file.
> >>>
> >>> 2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dl...@gmail.com>:
> >>>
> >>>> Usually map reduce output format would issue a much clearer
> >> message.this
> >>> on
> >>>> the other hand looks like a name node operation conflict.
> >>>>
> >>>> Sent from my phone.
> >>>> On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
> >>>>
> >>>>> Have u tried running with -ow (overwrite) option, that should clear
> >> all
> >>>>> tmpdir between successive runs ??
> >>>>>
> >>>>> The SSVD code does clear the tmpdir when -ow is specified.
> >>>>>
> >>>>> On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
> >>>>>
> >>>>>> we are running mahout ssvd, with a --tempDir parameter,
> >>>>>> but we are constantly getting this error below.
> >>>>>>
> >>>>>> the following seems to suggest that the 2 attempts in mapper are
> >>> trying
> >>>>> to
> >>>>>> access the same tmp dir on hdfs.
> >>>>>>
> >>>>>> if this is the cause, then ssvd has a bug in that different
> >> attempts
> >>>>> should
> >>>>>> use different temp dirs . right?
> >>>>>>
> >>>>>>
> >>>>>> I'm trying now without the --tempDir param
> >>>>>>
> >>>>>> thanks
> >>>>>> Yang
> >>>>>>
> >>>>>> Error:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> >>>>>> Lease mismatch on
> >>>>> /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> >>>>>> owned by
> >> DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1
> >>>> but
> >>>>> is
> >>>>>> accessed by
> >>>> DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
> >>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> >>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> >>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> >>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
> >>>>>>
> >>>>>> java.security.AccessController.doPrivileged(Native Method) at
> >>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> >>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> >>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1363) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> >>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> >>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> >>>>>> at
> >>>>>
> >> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> >>>>>> at
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> >>>>>> at
> >>>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> >>>>>> at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> >>>>>> at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> >>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> >>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> >>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> >>>>>> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> >>>>>> java.security.AccessController.doPrivileged(Native Method) at
> >>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> >>>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>
>
>
RE: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Ken Krugler <kk...@transpac.com>.
The BtJob's BtMapper has some "interesting" logic in its setup routine, where it looks like it's creating a side-channel:
/*
* actually this is kind of dangerous because this routine thinks we need
* to create file name for our current job and this will use -m- so it's
* just serendipity we are calling it from the mapper too as the QJob did.
*/
Path qInputPath =
new Path(qJobPath, FileOutputFormat.getUniqueFile(context,
QJob.OUTPUT_QHAT,
""));
blockNum = context.getTaskAttemptID().getTaskID().getId();
SequenceFileValueIterator<DenseBlockWritable> qhatInput =
new SequenceFileValueIterator<DenseBlockWritable>(qInputPath,
true,
conf);
closeables.addFirst(qhatInput);
It wouldn't surprise me if this fails when speculative execution is turned on.
-- Ken
> From: Dmitriy Lyubimov
> Sent: October 9, 2014 11:31:42am PDT
> To: user@mahout.apache.org
> Subject: Re: SSVD: lease conflict due to 2 attempts using the same dir
>
> wow. good to know. however, it would seem to me like a bug in
> MultipleOutputs? either way, it doesn't seem anything to do with the Mahout
> code itself.
>
> On Thu, Oct 9, 2014 at 10:32 AM, Yang <te...@gmail.com> wrote:
>
>> yes, that's what I'm saying, I disabled speculative, it works for now (kind
>> of is a hack)
>>
>>
>> also yes, this is hadoop 2.0 with YARN
>>
>> this has nothing to do with overwrite mode. the 2 attempts are run
>> simultaneously because they are speculative runs
>>
>> On Wed, Oct 8, 2014 at 12:07 AM, Serega Sheypak <se...@gmail.com>
>> wrote:
>>
>>> owned by
>>> DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1
>>> *but is
>>> accessed by
>>> DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1*
>>> at
>>>
>>> Looks like two clients try to write to the same dir and they are trying
>> to
>>> write to the same file.
>>>
>>> 2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dl...@gmail.com>:
>>>
>>>> Usually map reduce output format would issue a much clearer
>> message.this
>>> on
>>>> the other hand looks like a name node operation conflict.
>>>>
>>>> Sent from my phone.
>>>> On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
>>>>
>>>>> Have u tried running with -ow (overwrite) option, that should clear
>> all
>>>>> tmpdir between successive runs ??
>>>>>
>>>>> The SSVD code does clear the tmpdir when -ow is specified.
>>>>>
>>>>> On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
>>>>>
>>>>>> we are running mahout ssvd, with a --tempDir parameter,
>>>>>> but we are constantly getting this error below.
>>>>>>
>>>>>> the following seems to suggest that the 2 attempts in mapper are
>>> trying
>>>>> to
>>>>>> access the same tmp dir on hdfs.
>>>>>>
>>>>>> if this is the cause, then ssvd has a bug in that different
>> attempts
>>>>> should
>>>>>> use different temp dirs . right?
>>>>>>
>>>>>>
>>>>>> I'm trying now without the --tempDir param
>>>>>>
>>>>>> thanks
>>>>>> Yang
>>>>>>
>>>>>> Error:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>>>>>> Lease mismatch on
>>>>> /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
>>>>>> owned by
>> DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1
>>>> but
>>>>> is
>>>>>> accessed by
>>>> DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
>>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
>>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
>>>>>>
>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
>>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1410) at
>>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1363) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
>>>>>> at
>>>>>
>> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
>>>>>> at
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
>>>>>> at
>>>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
>>>>>> at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
>>>>>> at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
>>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
>>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
>>>>>> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
>>>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>>>>>
>>>>>
>>>>
>>>
>>
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
wow. good to know. however, it would seem to me like a bug in
MultipleOutputs? either way, it doesn't seem anything to do with the Mahout
code itself.
On Thu, Oct 9, 2014 at 10:32 AM, Yang <te...@gmail.com> wrote:
> yes, that's what I'm saying, I disabled speculative, it works for now (kind
> of is a hack)
>
>
> also yes, this is hadoop 2.0 with YARN
>
> this has nothing to do with overwrite mode. the 2 attempts are run
> simultaneously because they are speculative runs
>
> On Wed, Oct 8, 2014 at 12:07 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > owned by
> > DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1
> > *but is
> > accessed by
> > DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1*
> > at
> >
> > Looks like two clients try to write to the same dir and they are trying
> to
> > write to the same file.
> >
> > 2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dl...@gmail.com>:
> >
> > > Usually map reduce output format would issue a much clearer
> message.this
> > on
> > > the other hand looks like a name node operation conflict.
> > >
> > > Sent from my phone.
> > > On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
> > >
> > > > Have u tried running with -ow (overwrite) option, that should clear
> all
> > > > tmpdir between successive runs ??
> > > >
> > > > The SSVD code does clear the tmpdir when -ow is specified.
> > > >
> > > > On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
> > > >
> > > > > we are running mahout ssvd, with a --tempDir parameter,
> > > > > but we are constantly getting this error below.
> > > > >
> > > > > the following seems to suggest that the 2 attempts in mapper are
> > trying
> > > > to
> > > > > access the same tmp dir on hdfs.
> > > > >
> > > > > if this is the cause, then ssvd has a bug in that different
> attempts
> > > > should
> > > > > use different temp dirs . right?
> > > > >
> > > > >
> > > > > I'm trying now without the --tempDir param
> > > > >
> > > > > thanks
> > > > > Yang
> > > > >
> > > > > Error:
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > > > Lease mismatch on
> > > > /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> > > > > owned by
> DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1
> > > but
> > > > is
> > > > > accessed by
> > > DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
> > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> > > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> > > > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> > > > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
> > > > >
> > > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> > > > > org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> > > > > org.apache.hadoop.ipc.Client.call(Client.java:1363) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > > > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> > > > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> > > > > at
> > > >
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> > > > > at
> > > org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> > > > > at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> > > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> > > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> > > > > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> > > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > > >
> > > >
> > >
> >
>
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Yang <te...@gmail.com>.
yes, that's what I'm saying, I disabled speculative, it works for now (kind
of is a hack)
also yes, this is hadoop 2.0 with YARN
this has nothing to do with overwrite mode. the 2 attempts are run
simultaneously because they are speculative runs
On Wed, Oct 8, 2014 at 12:07 AM, Serega Sheypak <se...@gmail.com>
wrote:
> owned by
> DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1
> *but is
> accessed by
> DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1*
> at
>
> Looks like two clients try to write to the same dir and they are trying to
> write to the same file.
>
> 2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dl...@gmail.com>:
>
> > Usually map reduce output format would issue a much clearer message.this
> on
> > the other hand looks like a name node operation conflict.
> >
> > Sent from my phone.
> > On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
> >
> > > Have u tried running with -ow (overwrite) option, that should clear all
> > > tmpdir between successive runs ??
> > >
> > > The SSVD code does clear the tmpdir when -ow is specified.
> > >
> > > On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
> > >
> > > > we are running mahout ssvd, with a --tempDir parameter,
> > > > but we are constantly getting this error below.
> > > >
> > > > the following seems to suggest that the 2 attempts in mapper are
> trying
> > > to
> > > > access the same tmp dir on hdfs.
> > > >
> > > > if this is the cause, then ssvd has a bug in that different attempts
> > > should
> > > > use different temp dirs . right?
> > > >
> > > >
> > > > I'm trying now without the --tempDir param
> > > >
> > > > thanks
> > > > Yang
> > > >
> > > > Error:
> > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > > Lease mismatch on
> > > /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> > > > owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1
> > but
> > > is
> > > > accessed by
> > DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
> > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> > > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> > > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
> > > >
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> > > > org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> > > > org.apache.hadoop.ipc.Client.call(Client.java:1363) at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606) at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> > > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> > > > at
> > > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> > > > at
> > org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> > > > at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
> > > >
> > > >
> > >
> >
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> > > > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > >
> > >
> >
>
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Serega Sheypak <se...@gmail.com>.
owned by DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1
*but is
accessed by DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1*
at
Looks like two clients try to write to the same dir and they are trying to
write to the same file.
2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dl...@gmail.com>:
> Usually map reduce output format would issue a much clearer message.this on
> the other hand looks like a name node operation conflict.
>
> Sent from my phone.
> On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
>
> > Have u tried running with -ow (overwrite) option, that should clear all
> > tmpdir between successive runs ??
> >
> > The SSVD code does clear the tmpdir when -ow is specified.
> >
> > On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
> >
> > > we are running mahout ssvd, with a --tempDir parameter,
> > > but we are constantly getting this error below.
> > >
> > > the following seems to suggest that the 2 attempts in mapper are trying
> > to
> > > access the same tmp dir on hdfs.
> > >
> > > if this is the cause, then ssvd has a bug in that different attempts
> > should
> > > use different temp dirs . right?
> > >
> > >
> > > I'm trying now without the --tempDir param
> > >
> > > thanks
> > > Yang
> > >
> > > Error:
> > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > Lease mismatch on
> > /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> > > owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1
> but
> > is
> > > accessed by
> DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
> > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> > > at
> > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> > > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
> > >
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> > > org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> > > org.apache.hadoop.ipc.Client.call(Client.java:1363) at
> > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606) at
> > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> > > at
> > >
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> > > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> > > at
> > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> > > at
> > >
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> > > at
> > >
> >
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> > > at
> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> > > at
> > >
> > >
> >
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> > > at
> > >
> > >
> >
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> > > at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
> > >
> > >
> >
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> > > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > >
> >
>
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Usually map reduce output format would issue a much clearer message.this on
the other hand looks like a name node operation conflict.
Sent from my phone.
On Oct 7, 2014 3:23 PM, "Suneel Marthi" <sm...@apache.org> wrote:
> Have u tried running with -ow (overwrite) option, that should clear all
> tmpdir between successive runs ??
>
> The SSVD code does clear the tmpdir when -ow is specified.
>
> On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
>
> > we are running mahout ssvd, with a --tempDir parameter,
> > but we are constantly getting this error below.
> >
> > the following seems to suggest that the 2 attempts in mapper are trying
> to
> > access the same tmp dir on hdfs.
> >
> > if this is the cause, then ssvd has a bug in that different attempts
> should
> > use different temp dirs . right?
> >
> >
> > I'm trying now without the --tempDir param
> >
> > thanks
> > Yang
> >
> > Error:
> >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > Lease mismatch on
> /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> > owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1 but
> is
> > accessed by DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1
> at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> > at
> >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> > at
> >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> > at
> >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
> >
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> > org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> > org.apache.hadoop.ipc.Client.call(Client.java:1363) at
> >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606) at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> > at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> > at
> >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> > at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> > at
> >
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> > at
> >
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> > at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> > at
> >
> >
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> > at
> >
> >
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> > at
> >
> >
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> > at
> >
> >
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> > at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
> >
> >
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> >
>
Re: SSVD: lease conflict due to 2 attempts using the same dir
Posted by Suneel Marthi <sm...@apache.org>.
Have u tried running with -ow (overwrite) option, that should clear all
tmpdir between successive runs ??
The SSVD code does clear the tmpdir when -ow is specified.
On Tue, Oct 7, 2014 at 5:55 PM, Yang <te...@gmail.com> wrote:
> we are running mahout ssvd, with a --tempDir parameter,
> but we are constantly getting this error below.
>
> the following seems to suggest that the 2 attempts in mapper are trying to
> access the same tmp dir on hdfs.
>
> if this is the cause, then ssvd has a bug in that different attempts should
> use different temp dirs . right?
>
>
> I'm trying now without the --tempDir param
>
> thanks
> Yang
>
> Error:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> Lease mismatch on /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000
> owned by DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1 but is
> accessed by DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1 at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647)
> at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at
>
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at
> org.apache.hadoop.ipc.Client.call(Client.java:1410) at
> org.apache.hadoop.ipc.Client.call(Client.java:1363) at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.complete(Unknown Source) at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404)
> at
>
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116)
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100)
> at
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279)
> at
>
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501)
> at
>
> org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
> at
>
> org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537)
> at
>
> org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193)
> at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at
>
> org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>