You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Pavan Sudheendra <pa...@gmail.com> on 2013/08/29 16:33:52 UTC
Is hadoop tread safe?
Hi all,
Is hadoop thread safe? Do mappers make use of threads in any chance? A
little bit of information on the way they execute in parallel would help me
out. Thanks.
Regards,
Pavan
Re: Is hadoop tread safe?
Posted by Dinkar Sitaram <di...@gmail.com>.
This comment (from
http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm
)
may also be relevant:
"Hadoop purposely does not run more than one task at the same time in one
JVM for isolation purposes. And in stand-alone (local) mode, only one JVM
is ever used. If you want to make use of your four cores, you should run in
pseudo-distributed mode, and increase the max number of concurrent tasks to
four. You can do this with the mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum properties."
On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <ha...@cloudera.com> wrote:
> Map tasks run in parallel spawned JVMs, so are isolated from one
> another at runtime. Use of static functions shouldn't affect you
> generally.
>
> Default Map I/O is single-threaded. If you plan to use
> multiple-threads, use MultiThreadedMapper for proper thread-safety.
>
> On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com>
> wrote:
> > No, I had written a huge Map Reduce program which talks with hbase and
> does
> > a lot of computing using it as a source as well as sink.. One of my
> > colleague saw my code and saw that I had used a lot of static function
> > instead of making use of proper OOP concepts.. He was telling me that it
> > shouldn't be the way I should go about doing it.. But my code works
> fine..
> > So, was wondering will I face any problem in the future because of this..
> > That's all.
> >
> > Regards,
> > Pavan
> >
> > On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> >>
> >> Mappers don't communicate with each other in traditional MapReduce. If
> you
> >> need something more MPI-ish then look to MPI over YARN or write your own
> >> YARN app.
> >>
> >> If you need multi-threading within the mapper then it is up to you as
> the
> >> java developer to make it thread safe. Use the concurrent libraries like
> >> anything else and Bob's your uncle. Having overly-complicated mappers
> can be
> >> difficult to manage however and it kind of misses the mark for MapReduce
> >> problems.
> >>
> >> Maybe if you expand on your use case a bit someone here can provide
> >> specific advice.
> >>
> >>
> >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> >>> little bit of information on the way they execute in parallel would
> help me
> >>> out. Thanks.
> >>>
> >>> Regards,
> >>> Pavan
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
>
>
> --
> Harsh J
>
Re: Is hadoop tread safe?
Posted by Dinkar Sitaram <di...@gmail.com>.
This comment (from
http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm
)
may also be relevant:
"Hadoop purposely does not run more than one task at the same time in one
JVM for isolation purposes. And in stand-alone (local) mode, only one JVM
is ever used. If you want to make use of your four cores, you should run in
pseudo-distributed mode, and increase the max number of concurrent tasks to
four. You can do this with the mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum properties."
On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <ha...@cloudera.com> wrote:
> Map tasks run in parallel spawned JVMs, so are isolated from one
> another at runtime. Use of static functions shouldn't affect you
> generally.
>
> Default Map I/O is single-threaded. If you plan to use
> multiple-threads, use MultiThreadedMapper for proper thread-safety.
>
> On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com>
> wrote:
> > No, I had written a huge Map Reduce program which talks with hbase and
> does
> > a lot of computing using it as a source as well as sink.. One of my
> > colleague saw my code and saw that I had used a lot of static function
> > instead of making use of proper OOP concepts.. He was telling me that it
> > shouldn't be the way I should go about doing it.. But my code works
> fine..
> > So, was wondering will I face any problem in the future because of this..
> > That's all.
> >
> > Regards,
> > Pavan
> >
> > On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> >>
> >> Mappers don't communicate with each other in traditional MapReduce. If
> you
> >> need something more MPI-ish then look to MPI over YARN or write your own
> >> YARN app.
> >>
> >> If you need multi-threading within the mapper then it is up to you as
> the
> >> java developer to make it thread safe. Use the concurrent libraries like
> >> anything else and Bob's your uncle. Having overly-complicated mappers
> can be
> >> difficult to manage however and it kind of misses the mark for MapReduce
> >> problems.
> >>
> >> Maybe if you expand on your use case a bit someone here can provide
> >> specific advice.
> >>
> >>
> >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> >>> little bit of information on the way they execute in parallel would
> help me
> >>> out. Thanks.
> >>>
> >>> Regards,
> >>> Pavan
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
>
>
> --
> Harsh J
>
Re: Is hadoop tread safe?
Posted by Dinkar Sitaram <di...@gmail.com>.
This comment (from
http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm
)
may also be relevant:
"Hadoop purposely does not run more than one task at the same time in one
JVM for isolation purposes. And in stand-alone (local) mode, only one JVM
is ever used. If you want to make use of your four cores, you should run in
pseudo-distributed mode, and increase the max number of concurrent tasks to
four. You can do this with the mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum properties."
On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <ha...@cloudera.com> wrote:
> Map tasks run in parallel spawned JVMs, so are isolated from one
> another at runtime. Use of static functions shouldn't affect you
> generally.
>
> Default Map I/O is single-threaded. If you plan to use
> multiple-threads, use MultiThreadedMapper for proper thread-safety.
>
> On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com>
> wrote:
> > No, I had written a huge Map Reduce program which talks with hbase and
> does
> > a lot of computing using it as a source as well as sink.. One of my
> > colleague saw my code and saw that I had used a lot of static function
> > instead of making use of proper OOP concepts.. He was telling me that it
> > shouldn't be the way I should go about doing it.. But my code works
> fine..
> > So, was wondering will I face any problem in the future because of this..
> > That's all.
> >
> > Regards,
> > Pavan
> >
> > On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> >>
> >> Mappers don't communicate with each other in traditional MapReduce. If
> you
> >> need something more MPI-ish then look to MPI over YARN or write your own
> >> YARN app.
> >>
> >> If you need multi-threading within the mapper then it is up to you as
> the
> >> java developer to make it thread safe. Use the concurrent libraries like
> >> anything else and Bob's your uncle. Having overly-complicated mappers
> can be
> >> difficult to manage however and it kind of misses the mark for MapReduce
> >> problems.
> >>
> >> Maybe if you expand on your use case a bit someone here can provide
> >> specific advice.
> >>
> >>
> >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> >>> little bit of information on the way they execute in parallel would
> help me
> >>> out. Thanks.
> >>>
> >>> Regards,
> >>> Pavan
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
>
>
> --
> Harsh J
>
Re: Is hadoop tread safe?
Posted by Dinkar Sitaram <di...@gmail.com>.
This comment (from
http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm
)
may also be relevant:
"Hadoop purposely does not run more than one task at the same time in one
JVM for isolation purposes. And in stand-alone (local) mode, only one JVM
is ever used. If you want to make use of your four cores, you should run in
pseudo-distributed mode, and increase the max number of concurrent tasks to
four. You can do this with the mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum properties."
On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <ha...@cloudera.com> wrote:
> Map tasks run in parallel spawned JVMs, so are isolated from one
> another at runtime. Use of static functions shouldn't affect you
> generally.
>
> Default Map I/O is single-threaded. If you plan to use
> multiple-threads, use MultiThreadedMapper for proper thread-safety.
>
> On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com>
> wrote:
> > No, I had written a huge Map Reduce program which talks with hbase and
> does
> > a lot of computing using it as a source as well as sink.. One of my
> > colleague saw my code and saw that I had used a lot of static function
> > instead of making use of proper OOP concepts.. He was telling me that it
> > shouldn't be the way I should go about doing it.. But my code works
> fine..
> > So, was wondering will I face any problem in the future because of this..
> > That's all.
> >
> > Regards,
> > Pavan
> >
> > On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> >>
> >> Mappers don't communicate with each other in traditional MapReduce. If
> you
> >> need something more MPI-ish then look to MPI over YARN or write your own
> >> YARN app.
> >>
> >> If you need multi-threading within the mapper then it is up to you as
> the
> >> java developer to make it thread safe. Use the concurrent libraries like
> >> anything else and Bob's your uncle. Having overly-complicated mappers
> can be
> >> difficult to manage however and it kind of misses the mark for MapReduce
> >> problems.
> >>
> >> Maybe if you expand on your use case a bit someone here can provide
> >> specific advice.
> >>
> >>
> >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> >>> little bit of information on the way they execute in parallel would
> help me
> >>> out. Thanks.
> >>>
> >>> Regards,
> >>> Pavan
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
>
>
> --
> Harsh J
>
Re: Is hadoop tread safe?
Posted by Harsh J <ha...@cloudera.com>.
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you
generally.
Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.
On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
>
> Regards,
> Pavan
>
> On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
>>
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>>
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>>
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>>
>>
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>
>> wrote:
>>>
>>> Hi all,
>>>
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>>
>>> Regards,
>>> Pavan
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
--
Harsh J
Re: Is hadoop tread safe?
Posted by Harsh J <ha...@cloudera.com>.
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you
generally.
Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.
On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
>
> Regards,
> Pavan
>
> On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
>>
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>>
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>>
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>>
>>
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>
>> wrote:
>>>
>>> Hi all,
>>>
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>>
>>> Regards,
>>> Pavan
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
--
Harsh J
Re: Is hadoop tread safe?
Posted by Harsh J <ha...@cloudera.com>.
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you
generally.
Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.
On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
>
> Regards,
> Pavan
>
> On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
>>
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>>
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>>
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>>
>>
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>
>> wrote:
>>>
>>> Hi all,
>>>
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>>
>>> Regards,
>>> Pavan
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
--
Harsh J
Re: Is hadoop tread safe?
Posted by Harsh J <ha...@cloudera.com>.
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you
generally.
Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.
On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pa...@gmail.com> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
>
> Regards,
> Pavan
>
> On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
>>
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>>
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>>
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>>
>>
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>
>> wrote:
>>>
>>> Hi all,
>>>
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>>
>>> Regards,
>>> Pavan
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
--
Harsh J
Re: Is hadoop tread safe?
Posted by Pavan Sudheendra <pa...@gmail.com>.
No, I had written a huge Map Reduce program which talks with hbase and does
a lot of computing using it as a source as well as sink.. One of my
colleague saw my code and saw that I had used a lot of static function
instead of making use of proper OOP concepts.. He was telling me that it
shouldn't be the way I should go about doing it.. But my code works fine..
So, was wondering will I face any problem in the future because of this..
That's all.
Regards,
Pavan
On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> Mappers don't communicate with each other in traditional MapReduce. If you
> need something more MPI-ish then look to MPI over YARN or write your own
> YARN app.
>
> If you need multi-threading within the mapper then it is up to you as the
> java developer to make it thread safe. Use the concurrent libraries like
> anything else and Bob's your uncle. Having overly-complicated mappers can
> be difficult to manage however and it kind of misses the mark for MapReduce
> problems.
>
> Maybe if you expand on your use case a bit someone here can provide
> specific advice.
>
>
> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
>
>> Hi all,
>>
>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>> little bit of information on the way they execute in parallel would help me
>> out. Thanks.
>>
>> Regards,
>> Pavan
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Pavan Sudheendra <pa...@gmail.com>.
No, I had written a huge Map Reduce program which talks with hbase and does
a lot of computing using it as a source as well as sink.. One of my
colleague saw my code and saw that I had used a lot of static function
instead of making use of proper OOP concepts.. He was telling me that it
shouldn't be the way I should go about doing it.. But my code works fine..
So, was wondering will I face any problem in the future because of this..
That's all.
Regards,
Pavan
On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> Mappers don't communicate with each other in traditional MapReduce. If you
> need something more MPI-ish then look to MPI over YARN or write your own
> YARN app.
>
> If you need multi-threading within the mapper then it is up to you as the
> java developer to make it thread safe. Use the concurrent libraries like
> anything else and Bob's your uncle. Having overly-complicated mappers can
> be difficult to manage however and it kind of misses the mark for MapReduce
> problems.
>
> Maybe if you expand on your use case a bit someone here can provide
> specific advice.
>
>
> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
>
>> Hi all,
>>
>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>> little bit of information on the way they execute in parallel would help me
>> out. Thanks.
>>
>> Regards,
>> Pavan
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Pavan Sudheendra <pa...@gmail.com>.
No, I had written a huge Map Reduce program which talks with hbase and does
a lot of computing using it as a source as well as sink.. One of my
colleague saw my code and saw that I had used a lot of static function
instead of making use of proper OOP concepts.. He was telling me that it
shouldn't be the way I should go about doing it.. But my code works fine..
So, was wondering will I face any problem in the future because of this..
That's all.
Regards,
Pavan
On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> Mappers don't communicate with each other in traditional MapReduce. If you
> need something more MPI-ish then look to MPI over YARN or write your own
> YARN app.
>
> If you need multi-threading within the mapper then it is up to you as the
> java developer to make it thread safe. Use the concurrent libraries like
> anything else and Bob's your uncle. Having overly-complicated mappers can
> be difficult to manage however and it kind of misses the mark for MapReduce
> problems.
>
> Maybe if you expand on your use case a bit someone here can provide
> specific advice.
>
>
> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
>
>> Hi all,
>>
>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>> little bit of information on the way they execute in parallel would help me
>> out. Thanks.
>>
>> Regards,
>> Pavan
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Pavan Sudheendra <pa...@gmail.com>.
No, I had written a huge Map Reduce program which talks with hbase and does
a lot of computing using it as a source as well as sink.. One of my
colleague saw my code and saw that I had used a lot of static function
instead of making use of proper OOP concepts.. He was telling me that it
shouldn't be the way I should go about doing it.. But my code works fine..
So, was wondering will I face any problem in the future because of this..
That's all.
Regards,
Pavan
On Aug 29, 2013 8:11 PM, "Adam Muise" <am...@hortonworks.com> wrote:
> Mappers don't communicate with each other in traditional MapReduce. If you
> need something more MPI-ish then look to MPI over YARN or write your own
> YARN app.
>
> If you need multi-threading within the mapper then it is up to you as the
> java developer to make it thread safe. Use the concurrent libraries like
> anything else and Bob's your uncle. Having overly-complicated mappers can
> be difficult to manage however and it kind of misses the mark for MapReduce
> problems.
>
> Maybe if you expand on your use case a bit someone here can provide
> specific advice.
>
>
> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
>
>> Hi all,
>>
>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>> little bit of information on the way they execute in parallel would help me
>> out. Thanks.
>>
>> Regards,
>> Pavan
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Adam Muise <am...@hortonworks.com>.
Mappers don't communicate with each other in traditional MapReduce. If you
need something more MPI-ish then look to MPI over YARN or write your own
YARN app.
If you need multi-threading within the mapper then it is up to you as the
java developer to make it thread safe. Use the concurrent libraries like
anything else and Bob's your uncle. Having overly-complicated mappers can
be difficult to manage however and it kind of misses the mark for MapReduce
problems.
Maybe if you expand on your use case a bit someone here can provide
specific advice.
On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
> Hi all,
>
> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> little bit of information on the way they execute in parallel would help me
> out. Thanks.
>
> Regards,
> Pavan
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Adam Muise <am...@hortonworks.com>.
Mappers don't communicate with each other in traditional MapReduce. If you
need something more MPI-ish then look to MPI over YARN or write your own
YARN app.
If you need multi-threading within the mapper then it is up to you as the
java developer to make it thread safe. Use the concurrent libraries like
anything else and Bob's your uncle. Having overly-complicated mappers can
be difficult to manage however and it kind of misses the mark for MapReduce
problems.
Maybe if you expand on your use case a bit someone here can provide
specific advice.
On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
> Hi all,
>
> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> little bit of information on the way they execute in parallel would help me
> out. Thanks.
>
> Regards,
> Pavan
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Adam Muise <am...@hortonworks.com>.
Mappers don't communicate with each other in traditional MapReduce. If you
need something more MPI-ish then look to MPI over YARN or write your own
YARN app.
If you need multi-threading within the mapper then it is up to you as the
java developer to make it thread safe. Use the concurrent libraries like
anything else and Bob's your uncle. Having overly-complicated mappers can
be difficult to manage however and it kind of misses the mark for MapReduce
problems.
Maybe if you expand on your use case a bit someone here can provide
specific advice.
On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
> Hi all,
>
> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> little bit of information on the way they execute in parallel would help me
> out. Thanks.
>
> Regards,
> Pavan
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is hadoop tread safe?
Posted by Adam Muise <am...@hortonworks.com>.
Mappers don't communicate with each other in traditional MapReduce. If you
need something more MPI-ish then look to MPI over YARN or write your own
YARN app.
If you need multi-threading within the mapper then it is up to you as the
java developer to make it thread safe. Use the concurrent libraries like
anything else and Bob's your uncle. Having overly-complicated mappers can
be difficult to manage however and it kind of misses the mark for MapReduce
problems.
Maybe if you expand on your use case a bit someone here can provide
specific advice.
On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:
> Hi all,
>
> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> little bit of information on the way they execute in parallel would help me
> out. Thanks.
>
> Regards,
> Pavan
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.