File edit in Hadoop

[Stuti Awasthi <st...@hcl.com>]

Hi all,

Wish you all a very Happy New Year.

As I know that files once written in Hadoop are immutable. I can't edit the files in Apache distribution of Hadoop.
I tried with MapR distribution and I am able to edit, replace the files with this distribution.

I wanted to know does CDH3 also support file edit,replace functionality as normally done with NFS.

Please suggest.
Thanks


Regards,
Stuti Awasthi
HCL Comnet Systems and Services Ltd
F-8/9 Basement, Sec-3,Noida.


________________________________
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------

RE: File edit in Hadoop

[Stuti Awasthi <st...@hcl.com>]

Thanks to everyone for all responses. I would like to comment few things. I am working on Apache Hadoop from quite few years now. This forum has always resolved my queries upfront.

@Todd: +1 for your response. It clears out my query.

@Arun C Murthy: Thanks for your response. I always asked relevant questions on this forum. If I have query regarding the Hadoop functionality then I think this is the best place where I can get hints to proceed in my way. Through my query I was not targeting any vendors like MapR, CDH or Hortonworks but just like a normal user of Apache Hadoop I wanted to clear out a simple question. I hope this clears out my intention of putting this question here.

Thanks
Stuti



-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, January 04, 2012 12:01 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: File edit in Hadoop

On Tue, Jan 3, 2012 at 4:11 AM, Stuti Awasthi <st...@hcl.com> wrote:
> As I know that files once written in Hadoop are immutable. I can't
> edit the files in Apache distribution of Hadoop.
>
> I tried with MapR distribution and I am able to edit, replace the
> files with this distribution.
>

MapR isn't a Hadoop distribution - it's an entirely different closed-source filesystem, so it has a different feature set.

>
>
> I wanted to know does CDH3 also support file edit,replace
> functionality as normally done with NFS.
>

Nope, CDH is based on the open source Apache release, so has the same feature set.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------

Re: File edit in Hadoop

[Todd Lipcon <to...@cloudera.com>]

On Tue, Jan 3, 2012 at 4:11 AM, Stuti Awasthi <st...@hcl.com> wrote:
> As I know that files once written in Hadoop are immutable. I can’t edit the
> files in Apache distribution of Hadoop.
>
> I tried with MapR distribution and I am able to edit, replace the files with
> this distribution.
>

MapR isn't a Hadoop distribution - it's an entirely different
closed-source filesystem, so it has a different feature set.

>
>
> I wanted to know does CDH3 also support file edit,replace functionality as
> normally done with NFS.
>

Nope, CDH is based on the open source Apache release, so has the same
feature set.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

Re: File edit in Hadoop

[Todd Papaioannou <dr...@gmail.com>]

Nice Freudian ;)

On Jan 3, 2012, at 2:56 PM, M. C. Srivas wrote:

> MapR is the only distribution that supports MapR

Re: File edit in Hadoop

["M. C. Srivas" <mc...@gmail.com>]

On Tue, Jan 3, 2012 at 4:11 AM, Stuti Awasthi <st...@hcl.com> wrote:

>  Hi all,****
>
> ** **
>
> Wish you all a very Happy New Year.****
>
> ** **
>
> As I know that files once written in Hadoop are immutable. I can’t edit
> the files in Apache distribution of Hadoop.****
>
> I tried with MapR distribution and I am able to edit, replace the files
> with this distribution.****
>
> ** **
>
> I wanted to know does CDH3 also support file edit,replace functionality as
> normally done with NFS.
>

HDFS  (whether it comes from Apache or vendors like Cloudera or Hortonworks
who try to resell it) is architecturally incapable of random read-write.
NFS support requires random read-write capability, and thus HDFS cannot
support NFS. This is the same reason why you've also noticed funky behavior
when you used the FUSE interface (re: your earlier posts).

I think if you want NFS and Hadoop together, MapR is the only distribution
that supports MapR.  Of course, there are many other advantages to using
MapR apart from just NFS, but that's another story.





> ****
>
> ** **
>
> Please suggest.****
>
> Thanks****
>
> ** **
>
> ** **
>
> Regards,****
>
> *Stuti Awasthi*
>
> HCL Comnet Systems and Services Ltd****
>
> F-8/9 Basement, Sec-3,Noida.****
>
> ** **
>
> ------------------------------
> ::DISCLAIMER::
>
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect
> the opinions of HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of
> this message without the prior written consent of the author of this
> e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and
> attachments please check them for viruses and defect.
>
>
> -----------------------------------------------------------------------------------------------------------------------
>

Re: File edit in Hadoop

[Alexander Lorenz <wg...@googlemail.com>]

+1 for Arun and Todd 

Alexander Lorenz
http://mapredit.blogspot.com

On Jan 4, 2012, at 9:40 AM, Todd Lipcon <to...@cloudera.com> wrote:

> On Wed, Jan 4, 2012 at 9:28 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Other than these technical discussions, I don't see why ASF lists should be
>> used to discuss or market products of _any_ vendor for several good reasons:
>> # ASF developer community cannot help users of vendor-specific products for
>> obvious reasons.
>> # It could also lead to vendors spreading FUD about the ASF project while
>> downplaying the deficiencies of their products for their own gain.
>> 
>> Of course, feel free to point this out to others yourself.
> 
> +1. If you see me or another Cloudera employee hawking our commercial
> products on-list, feel free to point it out. Occasionally a user
> brings up CDH, which is Apache licensed and based on the ASF source
> and patches available in ASF. I'll answer those questions generally
> with pointers back to the ASF-hosted JIRA instance. But you won't find
> me trying to sell our paid/closed-source software here.
> 
> -Todd
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

Re: File edit in Hadoop

[Todd Lipcon <to...@cloudera.com>]

On Wed, Jan 4, 2012 at 9:28 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Other than these technical discussions, I don't see why ASF lists should be
> used to discuss or market products of _any_ vendor for several good reasons:
> # ASF developer community cannot help users of vendor-specific products for
> obvious reasons.
> # It could also lead to vendors spreading FUD about the ASF project while
> downplaying the deficiencies of their products for their own gain.
>
> Of course, feel free to point this out to others yourself.

+1. If you see me or another Cloudera employee hawking our commercial
products on-list, feel free to point it out. Occasionally a user
brings up CDH, which is Apache licensed and based on the ASF source
and patches available in ASF. I'll answer those questions generally
with pointers back to the ASF-hosted JIRA instance. But you won't find
me trying to sell our paid/closed-source software here.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

Re: File edit in Hadoop

[Arun C Murthy <ac...@hortonworks.com>]

On Jan 3, 2012, at 7:52 PM, M. C. Srivas wrote:
> 
> On Tue, Jan 3, 2012 at 4:01 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Stuti - it's best to stick to questions about Apache Hadoop on *@hadoop.apache.org lists. The Apache Hadoop mailing lists exist to help users and developers of Apache Hadoop.
> 
> this is an open forum, and if questions come up about  CDH or MapR  (or Hortonworks) or any other vendor's software or anyone's research that does similar things as Apache Hadoop, it is quite fair to discuss it here.   Instead of trying to curtail such conversations, feel free to point out advantages and disadvantages of any such work ... we will all benefit.

Again, ASF lists exist to help users and developers of ASF software. 

If you want to participate as a user or help improve the software you are very welcome. If you want to provide constructive feedback on how we can improve ASF software you are more than welcome. 

Other than these technical discussions, I don't see why ASF lists should be used to discuss or market products of _any_ vendor for several good reasons:
# ASF developer community cannot help users of vendor-specific products for obvious reasons. 
# It could also lead to vendors spreading FUD about the ASF project while downplaying the deficiencies of their products for their own gain. 

Of course, feel free to point this out to others yourself.

Arun

Re: File edit in Hadoop

["M. C. Srivas" <mc...@gmail.com>]

On Tue, Jan 3, 2012 at 4:01 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Stuti - it's best to stick to questions about Apache Hadoop on
> *@hadoop.apache.org lists. The Apache Hadoop mailing lists exist to help
> users and developers of Apache Hadoop.
>
> For vendor specific questions use vendor specific lists  and rely on your
> own homework - Srivas has very little incentive to point out MapR doesn't
> support end-to-end security and so on, not completely unexpected.
>

I think Stuti's question was about editing files ... and not security.

Arun,

this is an open forum, and if questions come up about  CDH or MapR  (or
Hortonworks) or any other vendor's software or anyone's research that does
similar things as Apache Hadoop, it is quite fair to discuss it here.
Instead of trying to curtail such conversations, feel free to point out
advantages and disadvantages of any such work ... we will all benefit.

Let's also be honest ---   just like MapR or HCL or any other company
that's out there,  Hortonworks and Cloudera are in this to make serious
profits for their investors and shareholders.



>
> Arun
>
> On Jan 3, 2012, at 4:11 AM, Stuti Awasthi wrote:
>
> Hi all,****
> ** **
> Wish you all a very Happy New Year.****
> ** **
> As I know that files once written in Hadoop are immutable. I can’t edit
> the files in Apache distribution of Hadoop.****
> I tried with MapR distribution and I am able to edit, replace the files
> with this distribution.****
> ** **
> I wanted to know does CDH3 also support file edit,replace functionality as
> normally done with NFS.****
> ** **
> Please suggest.****
> Thanks****
> ** **
> ** **
> Regards,****
> *Stuti Awasthi*
> HCL Comnet Systems and Services Ltd****
> F-8/9 Basement, Sec-3,Noida.****
> ** **
>
> ------------------------------
> ::DISCLAIMER::
>
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect
> the opinions of HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of
> this message without the prior written consent of the author of this
> e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and
> attachments please check them for viruses and defect.
>
>
> -----------------------------------------------------------------------------------------------------------------------
>
>
>

Re: File edit in Hadoop

[Arun C Murthy <ac...@hortonworks.com>]

Stuti - it's best to stick to questions about Apache Hadoop on *@hadoop.apache.org lists. The Apache Hadoop mailing lists exist to help users and developers of Apache Hadoop.

For vendor specific questions use vendor specific lists  and rely on your own homework - Srivas has very little incentive to point out MapR doesn't support end-to-end security and so on, not completely unexpected.

Arun

On Jan 3, 2012, at 4:11 AM, Stuti Awasthi wrote:

> Hi all,
>  
> Wish you all a very Happy New Year.
>  
> As I know that files once written in Hadoop are immutable. I can’t edit the files in Apache distribution of Hadoop.
> I tried with MapR distribution and I am able to edit, replace the files with this distribution.
>  
> I wanted to know does CDH3 also support file edit,replace functionality as normally done with NFS.
>  
> Please suggest.
> Thanks
>  
>  
> Regards,
> Stuti Awasthi
> HCL Comnet Systems and Services Ltd
> F-8/9 Basement, Sec-3,Noida.
>  
> 
> ::DISCLAIMER::
> -----------------------------------------------------------------------------------------------------------------------
> 
> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
> this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender immediately. Before opening any mail and
> attachments please check them for viruses and defect.
> 
> -----------------------------------------------------------------------------------------------------------------------