You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Krishna Kumar <kr...@nechclst.in> on 2009/12/01 14:59:59 UTC
concurrent write [ is it defined and consistent ? ]
Dear all,
Can anybody tell me how concurrent write happens in HDFS ?
Thanks and Best Regards,
Krishna Kumar
Senior Storage Engineer
Why do we have to die? If we had to die, and everything is gone after
that, then nothing else matters on this earth - everything is temporary,
at least relative to me.
DISCLAIMER:
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and
intended
for the named recipient(s) only.
It shall not attach any liability on the originator or NECHCL or its
affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the
opinions of NECHCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of
this message without the prior written consent of the author of this e-mail is
strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. .
-----------------------------------------------------------------------------------------------------------------------
Re: concurrent write [ is it defined and consistent ? ]
Posted by Hong Tang <ht...@yahoo-inc.com>.
Atomic append is useful in contexts like transactional logging and
producer-consumer workload sharing systems. Such usage cases are not
covered by map-reduce programming model.
On Dec 1, 2009, at 8:02 AM, Owen O'Malley wrote:
>
> On Dec 1, 2009, at 6:30 AM, Brian Bockelman wrote:
>
>> The upcoming 0.21.x release will be the first to support appends
>> after the file is closed (still single-writer though). I'd
>> speculate that this lays out some ground work that is necessary for
>> multi-writer atomic appends, but I don't even know if atomic
>> appends are even on the roadmap.
>
> The short answer is that I don't think anyone has even thought
> deeply about it. I haven't heard anyone talking about doing it
> anytime soon.
>
> My personal inclination is that atomic append does very bad things
> to both the design of the file system and the user interface to the
> file system. Clearly they added atomic append to GFS before they had
> MapReduce. It seems like most applications would be better served by
> implementing in MapReduce rather than using atomic append anyways...
>
> -- Owen
Re: concurrent write [ is it defined and consistent ? ]
Posted by "Hitchcock, Andrew" <an...@amazon.com>.
This interview might be interesting to folks following this discussion:
http://queue.acm.org/detail.cfm?id=1594206
It talks about some of the design decisions behind GFS. I recommend the whole article, but here is a snippet from the section on append:
QUINLAN At the time, [RecordAppend] must have seemed like a good idea, but in retrospect I think the consensus is that it proved to be more painful than it was worth. It just doesn't meet the expectations people have of a file system, so they end up getting surprised. Then they had to figure out work-arounds.
MCKUSICK In retrospect, how would you handle this differently?
QUINLAN I think it makes more sense to have a single writer per file.
MCKUSICK All right, but what happens when you have multiple people wanting to append to a log?
QUINLAN You serialize the writes through a single process that can ensure the replicas are consistent.
Andrew
On 12/1/09 8:02 AM, "Owen O'Malley" <om...@apache.org> wrote:
On Dec 1, 2009, at 6:30 AM, Brian Bockelman wrote:
> The upcoming 0.21.x release will be the first to support appends
> after the file is closed (still single-writer though). I'd
> speculate that this lays out some ground work that is necessary for
> multi-writer atomic appends, but I don't even know if atomic appends
> are even on the roadmap.
The short answer is that I don't think anyone has even thought deeply
about it. I haven't heard anyone talking about doing it anytime soon.
My personal inclination is that atomic append does very bad things to
both the design of the file system and the user interface to the file
system. Clearly they added atomic append to GFS before they had
MapReduce. It seems like most applications would be better served by
implementing in MapReduce rather than using atomic append anyways...
-- Owen
Re: concurrent write [ is it defined and consistent ? ]
Posted by Owen O'Malley <om...@apache.org>.
On Dec 1, 2009, at 6:30 AM, Brian Bockelman wrote:
> The upcoming 0.21.x release will be the first to support appends
> after the file is closed (still single-writer though). I'd
> speculate that this lays out some ground work that is necessary for
> multi-writer atomic appends, but I don't even know if atomic appends
> are even on the roadmap.
The short answer is that I don't think anyone has even thought deeply
about it. I haven't heard anyone talking about doing it anytime soon.
My personal inclination is that atomic append does very bad things to
both the design of the file system and the user interface to the file
system. Clearly they added atomic append to GFS before they had
MapReduce. It seems like most applications would be better served by
implementing in MapReduce rather than using atomic append anyways...
-- Owen
Re: concurrent write [ is it defined and consistent ? ]
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Krishna,
This type of functionality is not available in HDFS, unfortunately.
The upcoming 0.21.x release will be the first to support appends after the file is closed (still single-writer though). I'd speculate that this lays out some ground work that is necessary for multi-writer atomic appends, but I don't even know if atomic appends are even on the roadmap.
Brian
On Dec 1, 2009, at 8:27 AM, Krishna Kumar wrote:
> Ok , I was just trying to search some record append atomic operation [
> which happens in GFS ? ] . Is there anything similar in HDFS ?
>
> Thanks and Best Regards,
> Krishna Kumar
> Senior Storage Engineer
> Why do we have to die? If we had to die, and everything is gone after
> that, then nothing else matters on this earth - everything is temporary,
> at least relative to me.
>
>
>
> -----Original Message-----
> From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
> Sent: Tuesday, December 01, 2009 7:49 PM
> To: common-user@hadoop.apache.org
> Subject: Re: concurrent write [ is it defined and consistent ? ]
>
> I double checked, and it appears that the answer you are looking for is
> in a funny section ("Data replication"):
>
> "Files in HDFS are write-once and have strictly one writer at any time.
> "
> "There is a plan to support appending-writes to files in the future."
>
> So, there are no concurrent writes, but there are future plans for
> appends (currently in trunk).
>
> Brian
>
> On Dec 1, 2009, at 8:16 AM, Krishna Kumar wrote:
>
>> It doesn't give answer to my query .
>>
>> Thanks and Best Regards,
>> Krishna Kumar
>> Senior Storage Engineer
>> Why do we have to die? If we had to die, and everything is gone after
>> that, then nothing else matters on this earth - everything is
> temporary,
>> at least relative to me.
>>
>>
>> -----Original Message-----
>> From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
>> Sent: Tuesday, December 01, 2009 7:32 PM
>> To: common-user@hadoop.apache.org
>> Subject: Re: concurrent write [ is it defined and consistent ? ]
>>
>> Hey Krishna,
>>
>> You might benefit from the HDFS design document:
>>
>> http://hadoop.apache.org/common/docs/current/hdfs_design.html
>>
>> Let us know if it is missing important information.
>>
>> Brian
>>
>> On Dec 1, 2009, at 7:59 AM, Krishna Kumar wrote:
>>
>>> Dear all,
>>>
>>>
>>>
>>> Can anybody tell me how concurrent write happens in HDFS ?
>>>
>>>
>>>
>>> Thanks and Best Regards,
>>>
>>> Krishna Kumar
>>>
>>> Senior Storage Engineer
>>>
>>> Why do we have to die? If we had to die, and everything is gone after
>>> that, then nothing else matters on this earth - everything is
>> temporary,
>>> at least relative to me.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> DISCLAIMER:
>>>
>>
> ------------------------------------------------------------------------
>> -----------------------------------------------
>>> The contents of this e-mail and any attachment(s) are confidential
> and
>>> intended
>>> for the named recipient(s) only.
>>> It shall not attach any liability on the originator or NECHCL or its
>>> affiliates. Any views or opinions presented in
>>> this email are solely those of the author and may not necessarily
>> reflect the
>>> opinions of NECHCL or its affiliates.
>>> Any form of reproduction, dissemination, copying, disclosure,
>> modification,
>>> distribution and / or publication of
>>> this message without the prior written consent of the author of this
>> e-mail is
>>> strictly prohibited. If you have
>>> received this email in error please delete it and notify the sender
>>> immediately. .
>>>
>>
> ------------------------------------------------------------------------
>> -----------------------------------------------
>>
>>
>>
>>
>> DISCLAIMER:
>>
> ------------------------------------------------------------------------
> -----------------------------------------------
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended
>> for the named recipient(s) only.
>> It shall not attach any liability on the originator or NECHCL or its
>> affiliates. Any views or opinions presented in
>> this email are solely those of the author and may not necessarily
> reflect the
>> opinions of NECHCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
> modification,
>> distribution and / or publication of
>> this message without the prior written consent of the author of this
> e-mail is
>> strictly prohibited. If you have
>> received this email in error please delete it and notify the sender
>> immediately. .
>>
> ------------------------------------------------------------------------
> -----------------------------------------------
>
>
>
>
> DISCLAIMER:
> -----------------------------------------------------------------------------------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended
> for the named recipient(s) only.
> It shall not attach any liability on the originator or NECHCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect the
> opinions of NECHCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of
> this message without the prior written consent of the author of this e-mail is
> strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. .
> -----------------------------------------------------------------------------------------------------------------------
RE: concurrent write [ is it defined and consistent ? ]
Posted by Krishna Kumar <kr...@nechclst.in>.
Ok , I was just trying to search some record append atomic operation [
which happens in GFS ? ] . Is there anything similar in HDFS ?
Thanks and Best Regards,
Krishna Kumar
Senior Storage Engineer
Why do we have to die? If we had to die, and everything is gone after
that, then nothing else matters on this earth - everything is temporary,
at least relative to me.
-----Original Message-----
From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
Sent: Tuesday, December 01, 2009 7:49 PM
To: common-user@hadoop.apache.org
Subject: Re: concurrent write [ is it defined and consistent ? ]
I double checked, and it appears that the answer you are looking for is
in a funny section ("Data replication"):
"Files in HDFS are write-once and have strictly one writer at any time.
"
"There is a plan to support appending-writes to files in the future."
So, there are no concurrent writes, but there are future plans for
appends (currently in trunk).
Brian
On Dec 1, 2009, at 8:16 AM, Krishna Kumar wrote:
> It doesn't give answer to my query .
>
> Thanks and Best Regards,
> Krishna Kumar
> Senior Storage Engineer
> Why do we have to die? If we had to die, and everything is gone after
> that, then nothing else matters on this earth - everything is
temporary,
> at least relative to me.
>
>
> -----Original Message-----
> From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
> Sent: Tuesday, December 01, 2009 7:32 PM
> To: common-user@hadoop.apache.org
> Subject: Re: concurrent write [ is it defined and consistent ? ]
>
> Hey Krishna,
>
> You might benefit from the HDFS design document:
>
> http://hadoop.apache.org/common/docs/current/hdfs_design.html
>
> Let us know if it is missing important information.
>
> Brian
>
> On Dec 1, 2009, at 7:59 AM, Krishna Kumar wrote:
>
>> Dear all,
>>
>>
>>
>> Can anybody tell me how concurrent write happens in HDFS ?
>>
>>
>>
>> Thanks and Best Regards,
>>
>> Krishna Kumar
>>
>> Senior Storage Engineer
>>
>> Why do we have to die? If we had to die, and everything is gone after
>> that, then nothing else matters on this earth - everything is
> temporary,
>> at least relative to me.
>>
>>
>>
>>
>>
>>
>>
>>
>> DISCLAIMER:
>>
>
------------------------------------------------------------------------
> -----------------------------------------------
>> The contents of this e-mail and any attachment(s) are confidential
and
>> intended
>> for the named recipient(s) only.
>> It shall not attach any liability on the originator or NECHCL or its
>> affiliates. Any views or opinions presented in
>> this email are solely those of the author and may not necessarily
> reflect the
>> opinions of NECHCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
> modification,
>> distribution and / or publication of
>> this message without the prior written consent of the author of this
> e-mail is
>> strictly prohibited. If you have
>> received this email in error please delete it and notify the sender
>> immediately. .
>>
>
------------------------------------------------------------------------
> -----------------------------------------------
>
>
>
>
> DISCLAIMER:
>
------------------------------------------------------------------------
-----------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended
> for the named recipient(s) only.
> It shall not attach any liability on the originator or NECHCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily
reflect the
> opinions of NECHCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
modification,
> distribution and / or publication of
> this message without the prior written consent of the author of this
e-mail is
> strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. .
>
------------------------------------------------------------------------
-----------------------------------------------
DISCLAIMER:
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and
intended
for the named recipient(s) only.
It shall not attach any liability on the originator or NECHCL or its
affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the
opinions of NECHCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of
this message without the prior written consent of the author of this e-mail is
strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. .
-----------------------------------------------------------------------------------------------------------------------
Re: concurrent write [ is it defined and consistent ? ]
Posted by Brian Bockelman <bb...@cse.unl.edu>.
I double checked, and it appears that the answer you are looking for is in a funny section ("Data replication"):
"Files in HDFS are write-once and have strictly one writer at any time. "
"There is a plan to support appending-writes to files in the future."
So, there are no concurrent writes, but there are future plans for appends (currently in trunk).
Brian
On Dec 1, 2009, at 8:16 AM, Krishna Kumar wrote:
> It doesn't give answer to my query .
>
> Thanks and Best Regards,
> Krishna Kumar
> Senior Storage Engineer
> Why do we have to die? If we had to die, and everything is gone after
> that, then nothing else matters on this earth - everything is temporary,
> at least relative to me.
>
>
> -----Original Message-----
> From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
> Sent: Tuesday, December 01, 2009 7:32 PM
> To: common-user@hadoop.apache.org
> Subject: Re: concurrent write [ is it defined and consistent ? ]
>
> Hey Krishna,
>
> You might benefit from the HDFS design document:
>
> http://hadoop.apache.org/common/docs/current/hdfs_design.html
>
> Let us know if it is missing important information.
>
> Brian
>
> On Dec 1, 2009, at 7:59 AM, Krishna Kumar wrote:
>
>> Dear all,
>>
>>
>>
>> Can anybody tell me how concurrent write happens in HDFS ?
>>
>>
>>
>> Thanks and Best Regards,
>>
>> Krishna Kumar
>>
>> Senior Storage Engineer
>>
>> Why do we have to die? If we had to die, and everything is gone after
>> that, then nothing else matters on this earth - everything is
> temporary,
>> at least relative to me.
>>
>>
>>
>>
>>
>>
>>
>>
>> DISCLAIMER:
>>
> ------------------------------------------------------------------------
> -----------------------------------------------
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended
>> for the named recipient(s) only.
>> It shall not attach any liability on the originator or NECHCL or its
>> affiliates. Any views or opinions presented in
>> this email are solely those of the author and may not necessarily
> reflect the
>> opinions of NECHCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
> modification,
>> distribution and / or publication of
>> this message without the prior written consent of the author of this
> e-mail is
>> strictly prohibited. If you have
>> received this email in error please delete it and notify the sender
>> immediately. .
>>
> ------------------------------------------------------------------------
> -----------------------------------------------
>
>
>
>
> DISCLAIMER:
> -----------------------------------------------------------------------------------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended
> for the named recipient(s) only.
> It shall not attach any liability on the originator or NECHCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect the
> opinions of NECHCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of
> this message without the prior written consent of the author of this e-mail is
> strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. .
> -----------------------------------------------------------------------------------------------------------------------
RE: concurrent write [ is it defined and consistent ? ]
Posted by Krishna Kumar <kr...@nechclst.in>.
It doesn't give answer to my query .
Thanks and Best Regards,
Krishna Kumar
Senior Storage Engineer
Why do we have to die? If we had to die, and everything is gone after
that, then nothing else matters on this earth - everything is temporary,
at least relative to me.
-----Original Message-----
From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
Sent: Tuesday, December 01, 2009 7:32 PM
To: common-user@hadoop.apache.org
Subject: Re: concurrent write [ is it defined and consistent ? ]
Hey Krishna,
You might benefit from the HDFS design document:
http://hadoop.apache.org/common/docs/current/hdfs_design.html
Let us know if it is missing important information.
Brian
On Dec 1, 2009, at 7:59 AM, Krishna Kumar wrote:
> Dear all,
>
>
>
> Can anybody tell me how concurrent write happens in HDFS ?
>
>
>
> Thanks and Best Regards,
>
> Krishna Kumar
>
> Senior Storage Engineer
>
> Why do we have to die? If we had to die, and everything is gone after
> that, then nothing else matters on this earth - everything is
temporary,
> at least relative to me.
>
>
>
>
>
>
>
>
> DISCLAIMER:
>
------------------------------------------------------------------------
-----------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended
> for the named recipient(s) only.
> It shall not attach any liability on the originator or NECHCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily
reflect the
> opinions of NECHCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
modification,
> distribution and / or publication of
> this message without the prior written consent of the author of this
e-mail is
> strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. .
>
------------------------------------------------------------------------
-----------------------------------------------
DISCLAIMER:
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and
intended
for the named recipient(s) only.
It shall not attach any liability on the originator or NECHCL or its
affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the
opinions of NECHCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of
this message without the prior written consent of the author of this e-mail is
strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. .
-----------------------------------------------------------------------------------------------------------------------
Re: concurrent write [ is it defined and consistent ? ]
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Krishna,
You might benefit from the HDFS design document:
http://hadoop.apache.org/common/docs/current/hdfs_design.html
Let us know if it is missing important information.
Brian
On Dec 1, 2009, at 7:59 AM, Krishna Kumar wrote:
> Dear all,
>
>
>
> Can anybody tell me how concurrent write happens in HDFS ?
>
>
>
> Thanks and Best Regards,
>
> Krishna Kumar
>
> Senior Storage Engineer
>
> Why do we have to die? If we had to die, and everything is gone after
> that, then nothing else matters on this earth - everything is temporary,
> at least relative to me.
>
>
>
>
>
>
>
>
> DISCLAIMER:
> -----------------------------------------------------------------------------------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended
> for the named recipient(s) only.
> It shall not attach any liability on the originator or NECHCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect the
> opinions of NECHCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of
> this message without the prior written consent of the author of this e-mail is
> strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. .
> -----------------------------------------------------------------------------------------------------------------------