You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Torsten Curdt <tc...@apache.org> on 2007/11/05 11:05:18 UTC
rename dir while writing
Is there anywhere documented the expected behavior of concurrent
changes in the filesystem? As an example:
Hdfs client C1 is slowly writing to "/path/a/file". Now another hdfs
client C2 renames "/path/a" to /path/b".
What happens? Will C1 continue to write but the file will be in "/
path/b" when it closes the file? Will C2 get an exception because C1
is writing to "/path/a" so it cannot be renamed? And so on...
I will probably run off an do some test myself ...but is this
documented somewhere? I couldn't find it.
cheers
--
Torsten
Re: rename dir while writing
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Nov 5, 2007, at 9:48 AM, Johnson, Jorgen wrote:
> I was under the impression that a file doesn't 'exist' in hdfs
> global namespace until it has been closed, and that appends are not
> supported (yet?). So this situation, at current time, should not
> be possible, correct?
Look at the first change in 0.15.0 (HADOOP-1708). *smile* Files are
now visible when they are created.
-- Owen
Re: rename dir while writing
Posted by Torsten Curdt <tc...@apache.org>.
Please note that I am talking about renaming the parent
directory ...not the file itself!!
cheers
--
Torsten
On 05.11.2007, at 18:48, Johnson, Jorgen wrote:
> I was under the impression that a file doesn't 'exist' in hdfs
> global namespace until it has been closed, and that appends are not
> supported (yet?). So this situation, at current time, should not
> be possible, correct?
>
> I remember seeing a jira issue where someone wanted to change it so
> files show up before they are closed, but I didn't catch what the
> resolution was for that issue. Am I missing something here?
>
> -jorgenj
>
> -----Original Message-----
> From: Owen O'Malley [mailto:oom@yahoo-inc.com]
> Sent: Monday, November 05, 2007 9:24 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: rename dir while writing
>
>
> On Nov 5, 2007, at 2:05 AM, Torsten Curdt wrote:
>
>> Is there anywhere documented the expected behavior of concurrent
>> changes in the filesystem? As an example:
>>
>> Hdfs client C1 is slowly writing to "/path/a/file". Now another
>> hdfs client C2 renames "/path/a" to /path/b".
>>
>> What happens? Will C1 continue to write but the file will be in "/
>> path/b" when it closes the file? Will C2 get an exception because
>> C1 is writing to "/path/a" so it cannot be renamed? And so on...
>>
>> I will probably run off an do some test myself ...but is this
>> documented somewhere? I couldn't find it.
>
> The short answer is that currently bad things happen. All of the hdfs
> protocols call files by name and so changing the name of the file
> while someone is writing to it will break the writer. In my opinion,
> it would be better if dfs.ClientProtocol.create returned a lease
> handle, which was used for the follow up addBlock, complete, and
> renewLease.
>
> -- Owen
RE: rename dir while writing
Posted by "Johnson, Jorgen" <jo...@amazon.com>.
I was under the impression that a file doesn't 'exist' in hdfs global namespace until it has been closed, and that appends are not supported (yet?). So this situation, at current time, should not be possible, correct?
I remember seeing a jira issue where someone wanted to change it so files show up before they are closed, but I didn't catch what the resolution was for that issue. Am I missing something here?
-jorgenj
-----Original Message-----
From: Owen O'Malley [mailto:oom@yahoo-inc.com]
Sent: Monday, November 05, 2007 9:24 AM
To: hadoop-user@lucene.apache.org
Subject: Re: rename dir while writing
On Nov 5, 2007, at 2:05 AM, Torsten Curdt wrote:
> Is there anywhere documented the expected behavior of concurrent
> changes in the filesystem? As an example:
>
> Hdfs client C1 is slowly writing to "/path/a/file". Now another
> hdfs client C2 renames "/path/a" to /path/b".
>
> What happens? Will C1 continue to write but the file will be in "/
> path/b" when it closes the file? Will C2 get an exception because
> C1 is writing to "/path/a" so it cannot be renamed? And so on...
>
> I will probably run off an do some test myself ...but is this
> documented somewhere? I couldn't find it.
The short answer is that currently bad things happen. All of the hdfs
protocols call files by name and so changing the name of the file
while someone is writing to it will break the writer. In my opinion,
it would be better if dfs.ClientProtocol.create returned a lease
handle, which was used for the follow up addBlock, complete, and
renewLease.
-- Owen
Re: rename dir while writing
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Nov 5, 2007, at 2:05 AM, Torsten Curdt wrote:
> Is there anywhere documented the expected behavior of concurrent
> changes in the filesystem? As an example:
>
> Hdfs client C1 is slowly writing to "/path/a/file". Now another
> hdfs client C2 renames "/path/a" to /path/b".
>
> What happens? Will C1 continue to write but the file will be in "/
> path/b" when it closes the file? Will C2 get an exception because
> C1 is writing to "/path/a" so it cannot be renamed? And so on...
>
> I will probably run off an do some test myself ...but is this
> documented somewhere? I couldn't find it.
The short answer is that currently bad things happen. All of the hdfs
protocols call files by name and so changing the name of the file
while someone is writing to it will break the writer. In my opinion,
it would be better if dfs.ClientProtocol.create returned a lease
handle, which was used for the follow up addBlock, complete, and
renewLease.
-- Owen
RE: rename dir while writing
Posted by dhruba Borthakur <dh...@yahoo-inc.com>.
Renaming the directory shows the same behavior as renaming the file. The
writer will encounter an Exception.
Thanks,
dhruba
-----Original Message-----
From: Torsten Curdt [mailto:tcurdt@apache.org]
Sent: Monday, November 05, 2007 10:33 AM
To: hadoop-user@lucene.apache.org
Subject: Re: rename dir while writing
What about renaming the directory the file is in? ...not the file
itself.
I would have expected that writing is only "applied" once the file is
getting closed. So in theory the final directory would not even have
to exist before the file is close. ...but that was just a wild
assumption.
Anyway - point is: this really needs to be documented somewhere!
cheers
--
Torsten
On 05.11.2007, at 18:58, dhruba Borthakur wrote:
> Starting with release 0.15, a file will appear in the namespace as
> soon
> as it is created. If a writer is writing to a file and another client
> renames it, then the original writer will get an IO exception either
> when it finished writing to the current block or when it closes the
> file.
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: Torsten Curdt [mailto:tcurdt@apache.org]
> Sent: Monday, November 05, 2007 2:05 AM
> To: hadoop-user@lucene.apache.org
> Subject: rename dir while writing
>
> Is there anywhere documented the expected behavior of concurrent
> changes in the filesystem? As an example:
>
> Hdfs client C1 is slowly writing to "/path/a/file". Now another hdfs
> client C2 renames "/path/a" to /path/b".
>
> What happens? Will C1 continue to write but the file will be in "/
> path/b" when it closes the file? Will C2 get an exception because C1
> is writing to "/path/a" so it cannot be renamed? And so on...
>
> I will probably run off an do some test myself ...but is this
> documented somewhere? I couldn't find it.
>
> cheers
> --
> Torsten
Re: rename dir while writing
Posted by Torsten Curdt <tc...@apache.org>.
What about renaming the directory the file is in? ...not the file
itself.
I would have expected that writing is only "applied" once the file is
getting closed. So in theory the final directory would not even have
to exist before the file is close. ...but that was just a wild
assumption.
Anyway - point is: this really needs to be documented somewhere!
cheers
--
Torsten
On 05.11.2007, at 18:58, dhruba Borthakur wrote:
> Starting with release 0.15, a file will appear in the namespace as
> soon
> as it is created. If a writer is writing to a file and another client
> renames it, then the original writer will get an IO exception either
> when it finished writing to the current block or when it closes the
> file.
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: Torsten Curdt [mailto:tcurdt@apache.org]
> Sent: Monday, November 05, 2007 2:05 AM
> To: hadoop-user@lucene.apache.org
> Subject: rename dir while writing
>
> Is there anywhere documented the expected behavior of concurrent
> changes in the filesystem? As an example:
>
> Hdfs client C1 is slowly writing to "/path/a/file". Now another hdfs
> client C2 renames "/path/a" to /path/b".
>
> What happens? Will C1 continue to write but the file will be in "/
> path/b" when it closes the file? Will C2 get an exception because C1
> is writing to "/path/a" so it cannot be renamed? And so on...
>
> I will probably run off an do some test myself ...but is this
> documented somewhere? I couldn't find it.
>
> cheers
> --
> Torsten
RE: rename dir while writing
Posted by dhruba Borthakur <dh...@yahoo-inc.com>.
Starting with release 0.15, a file will appear in the namespace as soon
as it is created. If a writer is writing to a file and another client
renames it, then the original writer will get an IO exception either
when it finished writing to the current block or when it closes the
file.
Thanks,
dhruba
-----Original Message-----
From: Torsten Curdt [mailto:tcurdt@apache.org]
Sent: Monday, November 05, 2007 2:05 AM
To: hadoop-user@lucene.apache.org
Subject: rename dir while writing
Is there anywhere documented the expected behavior of concurrent
changes in the filesystem? As an example:
Hdfs client C1 is slowly writing to "/path/a/file". Now another hdfs
client C2 renames "/path/a" to /path/b".
What happens? Will C1 continue to write but the file will be in "/
path/b" when it closes the file? Will C2 get an exception because C1
is writing to "/path/a" so it cannot be renamed? And so on...
I will probably run off an do some test myself ...but is this
documented somewhere? I couldn't find it.
cheers
--
Torsten