You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Balaraman, Anand" <An...@SYNTELINC.COM> on 2012/09/28 06:15:12 UTC
Can we write output directly to HDFS from Mapper
Hi
In Map-Reduce, is it appropriate to write the output directly to HDFS
from Mapper (without using a reducer) ?
Are there any adverse effects in doing so or are there any best
practices to be followed in this aspect ?
Comments are much appreciable at the moment J
Thanks and Regards
Anand B
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Re: Can we write output directly to HDFS from Mapper
Posted by Harsh J <ha...@cloudera.com>.
Anand,
You may read this in the FAQ:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Harsh J
Re: Can we write output directly to HDFS from Mapper
Posted by Harsh J <ha...@cloudera.com>.
Anand,
You may read this in the FAQ:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Harsh J
Re: Can we write output directly to HDFS from Mapper
Posted by Hemanth Yamijala <yh...@gmail.com>.
Can certainly do that. Indeed, if you set the number of reducers to 0,
the map output will be directly written to HDFS by the framework
itself. You may also want to look at
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files
to see some things that need to be taken care of if you are writing
files on your own.
Thanks
hemanth
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
Re: Can we write output directly to HDFS from Mapper
Posted by Hemanth Yamijala <yh...@gmail.com>.
Can certainly do that. Indeed, if you set the number of reducers to 0,
the map output will be directly written to HDFS by the framework
itself. You may also want to look at
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files
to see some things that need to be taken care of if you are writing
files on your own.
Thanks
hemanth
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
Re: Can we write output directly to HDFS from Mapper
Posted by Harsh J <ha...@cloudera.com>.
Anand,
You may read this in the FAQ:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Harsh J
Re: Can we write output directly to HDFS from Mapper
Posted by Harsh J <ha...@cloudera.com>.
Anand,
You may read this in the FAQ:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Harsh J
Re: Can we write output directly to HDFS from Mapper
Posted by Hemanth Yamijala <yh...@gmail.com>.
Can certainly do that. Indeed, if you set the number of reducers to 0,
the map output will be directly written to HDFS by the framework
itself. You may also want to look at
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files
to see some things that need to be taken care of if you are writing
files on your own.
Thanks
hemanth
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
Re: Can we write output directly to HDFS from Mapper
Posted by Harsh J <ha...@cloudera.com>.
Anand,
You may read this in the FAQ:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Harsh J
Re: Can we write output directly to HDFS from Mapper
Posted by Hemanth Yamijala <yh...@gmail.com>.
Can certainly do that. Indeed, if you set the number of reducers to 0,
the map output will be directly written to HDFS by the framework
itself. You may also want to look at
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files
to see some things that need to be taken care of if you are writing
files on your own.
Thanks
hemanth
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
Re: Can we write output directly to HDFS from Mapper
Posted by Hemanth Yamijala <yh...@gmail.com>.
Can certainly do that. Indeed, if you set the number of reducers to 0,
the map output will be directly written to HDFS by the framework
itself. You may also want to look at
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files
to see some things that need to be taken care of if you are writing
files on your own.
Thanks
hemanth
On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand
<An...@syntelinc.com> wrote:
> Hi
>
>
>
> In Map-Reduce, is it appropriate to write the output directly to HDFS from
> Mapper (without using a reducer) ?
>
> Are there any adverse effects in doing so or are there any best practices to
> be followed in this aspect ?
>
>
>
> Comments are much appreciable at the moment J
>
>
>
> Thanks and Regards
>
> Anand B
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.