You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ahmad Humayun <ah...@gmail.com> on 2008/03/12 14:07:50 UTC
deleting files from the DFS
Hi there,
I have some questions about the DFS functionality.
Case 1: Lets say there is a node which opens a file on the DFS to read. From
my understanding, the open request will go to the namenode, which will tell
it where the blocks of the file are. What if another node comes up and asks
to delete that file. Of course this request will go to the namenode. How
will the namenode respond in such a situation?
Case 2: Whenever a node wants to delete a file, I think the request first
lands with the namenode? What happens to the replicas of the blocks of that
file, as soon as that file gets deleted?
Thank you for all the help :)
regards,
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: deleting files from the DFS
Posted by Ahmad Humayun <ah...@gmail.com>.
Yes!! So is there any locking mechanisms? Plus what dictates this delay in
file deletion and actual deletion of the file blocks over the cluster?
regards,
On Fri, Mar 14, 2008 at 6:22 AM, edward yoon <ed...@udanax.org> wrote:
> Can zookeeper locking services be used for these cases?
>
> Thanks,
> Edward.
>
> On 3/13/08, Sanjay Radia <sr...@yahoo-inc.com> wrote:
> > Ahmad Humayun wrote:
> > > So does that mean nodes can possibly read files that have been
> "deleted"
> > >
> > If the name node entry has been deleted, new opens will fail.
> > Already opened files may have their reads succeed if they have cached
> > the block locations and also if the
> > blocks have not been deleted.
> >
> > Hence, if you open a large file and someone deletes the file after you
> > open it, at some point the reads may fail if
> > the client needs to go to the namenode to get block locations.
> > This not posix semantics but keeping the namenode stateless has
> > advantages in scaling.
> > > thanks :)
> > >
> > >
> > > regards,
> > >
> > > On Wed, Mar 12, 2008 at 10:57 PM, Hairong Kuang <hairong@yahoo-inc.com
> >
> > > wrote:
> > >
> > >
> > >>> Case 1: Lets say there is a node which opens a file on the DFS to
> read.
> > >>>
> > >> From
> > >>
> > >>> my understanding, the open request will go to the namenode, which
> will
> > >>>
> > >> tell
> > >>
> > >>> it where the blocks of the file are. What if another node comes up
> and
> > >>>
> > >> asks
> > >>
> > >>> to delete that file. Of course this request will go to the namenode.
> How
> > >>> will the namenode respond in such a situation?
> > >>>
> > >>>
> > >> The namenode is stateless. So the delete request will go through. The
> > >> first
> > >> request may fail.
> > >>
> > >>
> > >>> Case 2: Whenever a node wants to delete a file, I think the request
> > >>>
> > >> first
> > >>
> > >>> lands with the namenode? What happens to the replicas of the blocks
> of
> > >>>
> > >> that
> > >>
> > >>> file, as soon as that file gets deleted?
> > >>>
> > >>>
> > >> The replicas are scheduled to be deleted by the namenode. But there
> may be
> > >> some delay before they are actually deleted on the datanodes.
> > >>
> > >> Hairong
> > >>
> > >>
> > >>
> > >
> > >
> > >
> >
> >
>
>
> --
> B. Regards,
> Edward yoon @ NHN, corp.
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: deleting files from the DFS
Posted by edward yoon <ed...@udanax.org>.
Can zookeeper locking services be used for these cases?
Thanks,
Edward.
On 3/13/08, Sanjay Radia <sr...@yahoo-inc.com> wrote:
> Ahmad Humayun wrote:
> > So does that mean nodes can possibly read files that have been "deleted"
> >
> If the name node entry has been deleted, new opens will fail.
> Already opened files may have their reads succeed if they have cached
> the block locations and also if the
> blocks have not been deleted.
>
> Hence, if you open a large file and someone deletes the file after you
> open it, at some point the reads may fail if
> the client needs to go to the namenode to get block locations.
> This not posix semantics but keeping the namenode stateless has
> advantages in scaling.
> > thanks :)
> >
> >
> > regards,
> >
> > On Wed, Mar 12, 2008 at 10:57 PM, Hairong Kuang <ha...@yahoo-inc.com>
> > wrote:
> >
> >
> >>> Case 1: Lets say there is a node which opens a file on the DFS to read.
> >>>
> >> From
> >>
> >>> my understanding, the open request will go to the namenode, which will
> >>>
> >> tell
> >>
> >>> it where the blocks of the file are. What if another node comes up and
> >>>
> >> asks
> >>
> >>> to delete that file. Of course this request will go to the namenode. How
> >>> will the namenode respond in such a situation?
> >>>
> >>>
> >> The namenode is stateless. So the delete request will go through. The
> >> first
> >> request may fail.
> >>
> >>
> >>> Case 2: Whenever a node wants to delete a file, I think the request
> >>>
> >> first
> >>
> >>> lands with the namenode? What happens to the replicas of the blocks of
> >>>
> >> that
> >>
> >>> file, as soon as that file gets deleted?
> >>>
> >>>
> >> The replicas are scheduled to be deleted by the namenode. But there may be
> >> some delay before they are actually deleted on the datanodes.
> >>
> >> Hairong
> >>
> >>
> >>
> >
> >
> >
>
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: deleting files from the DFS
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
Ahmad Humayun wrote:
> So does that mean nodes can possibly read files that have been "deleted"
>
If the name node entry has been deleted, new opens will fail.
Already opened files may have their reads succeed if they have cached
the block locations and also if the
blocks have not been deleted.
Hence, if you open a large file and someone deletes the file after you
open it, at some point the reads may fail if
the client needs to go to the namenode to get block locations.
This not posix semantics but keeping the namenode stateless has
advantages in scaling.
> thanks :)
>
>
> regards,
>
> On Wed, Mar 12, 2008 at 10:57 PM, Hairong Kuang <ha...@yahoo-inc.com>
> wrote:
>
>
>>> Case 1: Lets say there is a node which opens a file on the DFS to read.
>>>
>> From
>>
>>> my understanding, the open request will go to the namenode, which will
>>>
>> tell
>>
>>> it where the blocks of the file are. What if another node comes up and
>>>
>> asks
>>
>>> to delete that file. Of course this request will go to the namenode. How
>>> will the namenode respond in such a situation?
>>>
>>>
>> The namenode is stateless. So the delete request will go through. The
>> first
>> request may fail.
>>
>>
>>> Case 2: Whenever a node wants to delete a file, I think the request
>>>
>> first
>>
>>> lands with the namenode? What happens to the replicas of the blocks of
>>>
>> that
>>
>>> file, as soon as that file gets deleted?
>>>
>>>
>> The replicas are scheduled to be deleted by the namenode. But there may be
>> some delay before they are actually deleted on the datanodes.
>>
>> Hairong
>>
>>
>>
>
>
>
Re: deleting files from the DFS
Posted by Ahmad Humayun <ah...@gmail.com>.
So does that mean nodes can possibly read files that have been "deleted"
thanks :)
regards,
On Wed, Mar 12, 2008 at 10:57 PM, Hairong Kuang <ha...@yahoo-inc.com>
wrote:
> > Case 1: Lets say there is a node which opens a file on the DFS to read.
> From
> > my understanding, the open request will go to the namenode, which will
> tell
> > it where the blocks of the file are. What if another node comes up and
> asks
> > to delete that file. Of course this request will go to the namenode. How
> > will the namenode respond in such a situation?
> >
>
> The namenode is stateless. So the delete request will go through. The
> first
> request may fail.
>
> > Case 2: Whenever a node wants to delete a file, I think the request
> first
> > lands with the namenode? What happens to the replicas of the blocks of
> that
> > file, as soon as that file gets deleted?
> >
>
> The replicas are scheduled to be deleted by the namenode. But there may be
> some delay before they are actually deleted on the datanodes.
>
> Hairong
>
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: deleting files from the DFS
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
> Case 1: Lets say there is a node which opens a file on the DFS to read. From
> my understanding, the open request will go to the namenode, which will tell
> it where the blocks of the file are. What if another node comes up and asks
> to delete that file. Of course this request will go to the namenode. How
> will the namenode respond in such a situation?
>
The namenode is stateless. So the delete request will go through. The first
request may fail.
> Case 2: Whenever a node wants to delete a file, I think the request first
> lands with the namenode? What happens to the replicas of the blocks of that
> file, as soon as that file gets deleted?
>
The replicas are scheduled to be deleted by the namenode. But there may be
some delay before they are actually deleted on the datanodes.
Hairong