You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Saikat Kanjilal <sx...@hotmail.com> on 2011/09/09 17:58:37 UTC
Research projects for hadoop
Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace
I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
Regards
RE: Research projects for hadoop
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Thanks all for your feedback on this, I'll research some more and gather some benchmarks, also what about the provisioning of long running services via HOD, is anyone looking at this project or has done work on this?
Regards
> From: msegel@navteq.com
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Date: Fri, 9 Sep 2011 12:45:38 -0500
> Subject: RE: Research projects for hadoop
>
> Why would you want to take a perfectly good machine and then try to virtualize it?
> I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks. However if I virtualize the box, I lose at least 1 core per VM so I end up with 4 nodes that have less capabilities and performance than I would have under my original box....
>
>
> -----Original Message-----
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> Sent: Friday, September 09, 2011 10:59 AM
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Research projects for hadoop
>
>
> Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
> Regards
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Research projects for hadoop
Posted by Andrew Purtell <ap...@apache.org>.
Both Hadoop and virtualization are means to an end. That end is to consolidate workloads traditionally deployed to separate servers so the average utilization and ROI of a given server increases.
Companies looking to consolidate data-intensive computation may be better served moving to Hadoop infrastructure than a virtualization project. Let me give you an example:
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> By assigning a virtual machine to a datanode, we effectively isolate
> the datanode from the load on the machine caused by other processes, making the
> datanode more responsive/reliable.W
One can set up virtual partitions of CPU and RAM resources that can be fairly independent, but attempting to stack I/O intensive workloads on top of each other via virtualization is a recipe for lower performance, negative ROI, and dissatisfied users.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: "Segel, Mike" <ms...@navteq.com>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Cc:
> Sent: Friday, September 9, 2011 10:45 AM
> Subject: RE: Research projects for hadoop
>
> Why would you want to take a perfectly good machine and then try to virtualize
> it?
> I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks.
> However if I virtualize the box, I lose at least 1 core per VM so I end up with
> 4 nodes that have less capabilities and performance than I would have under my
> original box....
>
>
> -----Original Message-----
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> Sent: Friday, September 09, 2011 10:59 AM
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Research projects for hadoop
>
>
> Hi Folks,I was looking through the following wiki page:
> http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if
> there's been any work done (or any interest to do work) for the following
> topics:
> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one
> integrate sandboxing of arbitrary user code in C++ and other languages in a VM
> such as Xen with the Hadoop framework? How does this interact with SGE, Torque,
> Condor?As each individual machine has more and more cores/cpus, it makes sense
> to partition each machine into multiple virtual machines. That gives us a number
> of benefits:By assigning a virtual machine to a datanode, we effectively isolate
> the datanode from the load on the machine caused by other processes, making the
> datanode more responsive/reliable.With multiple virtual machines on each
> machine, we can lower the granularity of hod scheduling units, making it
> possible to schedule multiple tasktrackers on the same machine, improving the
> overall utilization of the whole clusters.With virtualization, we can easily
> snapshot a virtual cluster before releasing it, making it possible to
> re-activate the same cluster in the future and start to work from the
> snapshot.Provisioning of long running Services via HODWork on a computation
> model for services on the grid. The model would include:Various tools for
> defining clients and servers of the service, and at the least a C++ and Java
> instantiation of the abstractionsLogical definitions of how to partition work
> onto a set of servers, i.e. a generalized shard implementationA few useful
> abstractions like locks (exclusive and RW, fairness), leader election,
> transactions,Various communication models for groups of servers belonging to a
> service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability,
> managing pools of servers for a service with spares, etc.Integration with HDFS
> for persistence, as well as access to local filesystemsIntegration with
> ZooKeeper so that applications can use the namespace I would like to either help
> out with a design for the above or prototyping code, please let me know if and
> what the process may be to move forward with this.
> Regards
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not the
> intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error, please
> notify the sender and delete/destroy the original message and any copy of it
> from your computer or paper files.
>
Re: Research projects for hadoop
Posted by Andrew Purtell <ap...@apache.org>.
Both Hadoop and virtualization are means to an end. That end is to consolidate workloads traditionally deployed to separate servers so the average utilization and ROI of a given server increases.
Companies looking to consolidate data-intensive computation may be better served moving to Hadoop infrastructure than a virtualization project. Let me give you an example:
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> By assigning a virtual machine to a datanode, we effectively isolate
> the datanode from the load on the machine caused by other processes, making the
> datanode more responsive/reliable.W
One can set up virtual partitions of CPU and RAM resources that can be fairly independent, but attempting to stack I/O intensive workloads on top of each other via virtualization is a recipe for lower performance, negative ROI, and dissatisfied users.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: "Segel, Mike" <ms...@navteq.com>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Cc:
> Sent: Friday, September 9, 2011 10:45 AM
> Subject: RE: Research projects for hadoop
>
> Why would you want to take a perfectly good machine and then try to virtualize
> it?
> I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks.
> However if I virtualize the box, I lose at least 1 core per VM so I end up with
> 4 nodes that have less capabilities and performance than I would have under my
> original box....
>
>
> -----Original Message-----
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> Sent: Friday, September 09, 2011 10:59 AM
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Research projects for hadoop
>
>
> Hi Folks,I was looking through the following wiki page:
> http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if
> there's been any work done (or any interest to do work) for the following
> topics:
> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one
> integrate sandboxing of arbitrary user code in C++ and other languages in a VM
> such as Xen with the Hadoop framework? How does this interact with SGE, Torque,
> Condor?As each individual machine has more and more cores/cpus, it makes sense
> to partition each machine into multiple virtual machines. That gives us a number
> of benefits:By assigning a virtual machine to a datanode, we effectively isolate
> the datanode from the load on the machine caused by other processes, making the
> datanode more responsive/reliable.With multiple virtual machines on each
> machine, we can lower the granularity of hod scheduling units, making it
> possible to schedule multiple tasktrackers on the same machine, improving the
> overall utilization of the whole clusters.With virtualization, we can easily
> snapshot a virtual cluster before releasing it, making it possible to
> re-activate the same cluster in the future and start to work from the
> snapshot.Provisioning of long running Services via HODWork on a computation
> model for services on the grid. The model would include:Various tools for
> defining clients and servers of the service, and at the least a C++ and Java
> instantiation of the abstractionsLogical definitions of how to partition work
> onto a set of servers, i.e. a generalized shard implementationA few useful
> abstractions like locks (exclusive and RW, fairness), leader election,
> transactions,Various communication models for groups of servers belonging to a
> service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability,
> managing pools of servers for a service with spares, etc.Integration with HDFS
> for persistence, as well as access to local filesystemsIntegration with
> ZooKeeper so that applications can use the namespace I would like to either help
> out with a design for the above or prototyping code, please let me know if and
> what the process may be to move forward with this.
> Regards
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not the
> intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error, please
> notify the sender and delete/destroy the original message and any copy of it
> from your computer or paper files.
>
RE: Research projects for hadoop
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Thanks all for your feedback on this, I'll research some more and gather some benchmarks, also what about the provisioning of long running services via HOD, is anyone looking at this project or has done work on this?
Regards
> From: msegel@navteq.com
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Date: Fri, 9 Sep 2011 12:45:38 -0500
> Subject: RE: Research projects for hadoop
>
> Why would you want to take a perfectly good machine and then try to virtualize it?
> I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks. However if I virtualize the box, I lose at least 1 core per VM so I end up with 4 nodes that have less capabilities and performance than I would have under my original box....
>
>
> -----Original Message-----
> From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
> Sent: Friday, September 09, 2011 10:59 AM
> To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Research projects for hadoop
>
>
> Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
> Regards
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
RE: Research projects for hadoop
Posted by "Segel, Mike" <ms...@navteq.com>.
Why would you want to take a perfectly good machine and then try to virtualize it?
I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks. However if I virtualize the box, I lose at least 1 core per VM so I end up with 4 nodes that have less capabilities and performance than I would have under my original box....
-----Original Message-----
From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
Sent: Friday, September 09, 2011 10:59 AM
To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Research projects for hadoop
Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
Regards
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
RE: Research projects for hadoop
Posted by "Segel, Mike" <ms...@navteq.com>.
Why would you want to take a perfectly good machine and then try to virtualize it?
I mean if I have 4 quad core cpus, I can run a lot of simultaneous map tasks. However if I virtualize the box, I lose at least 1 core per VM so I end up with 4 nodes that have less capabilities and performance than I would have under my original box....
-----Original Message-----
From: Saikat Kanjilal [mailto:sxk1969@hotmail.com]
Sent: Friday, September 09, 2011 10:59 AM
To: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Research projects for hadoop
Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
Regards
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Research projects for hadoop
Posted by Arun C Murthy <ac...@hortonworks.com>.
Saikat,
As Robert pointed out, performance is a primary criterion - maybe you can come back with benchmarks? Try sorts with >100G data.
Also, MRv2 makes it easy to play with these, you might want to try that.
Arun
On Sep 9, 2011, at 10:34 AM, Saikat Kanjilal wrote:
>
> How about using virtual box and centos 64 bit to serve as a linux container for isolating map/reduce processes? I have setup this up in the past, its really easy.
>
>
>> From: evans@yahoo-inc.com
>> To: mapreduce-dev@hadoop.apache.org
>> Date: Fri, 9 Sep 2011 10:30:37 -0700
>> Subject: Re: Research projects for hadoop
>>
>> The biggest issue with Xen and other virtualization technologies is that often there is an IO penalty involved with using them. For many jobs this is not an acceptable trade off. I do know, however, that there has been some discussion about using Linux Containers for isolation of Map/Reduce processes. I don't know if any JIRA has been filed for it or not, but they are much lighter weight then Xen and other virtualization tech, because all it really is concerned with is resource isolation, and not virtualizing an entire operating system.
>>
>> --Bobby Evans
>>
>> On 9/9/11 10:58 AM, "Saikat Kanjilal" <sx...@hotmail.com> wrote:
>>
>>
>>
>> Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
>> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace
>> I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
>> Regards
>>
>
RE: Research projects for hadoop
Posted by Saikat Kanjilal <sx...@hotmail.com>.
How about using virtual box and centos 64 bit to serve as a linux container for isolating map/reduce processes? I have setup this up in the past, its really easy.
> From: evans@yahoo-inc.com
> To: mapreduce-dev@hadoop.apache.org
> Date: Fri, 9 Sep 2011 10:30:37 -0700
> Subject: Re: Research projects for hadoop
>
> The biggest issue with Xen and other virtualization technologies is that often there is an IO penalty involved with using them. For many jobs this is not an acceptable trade off. I do know, however, that there has been some discussion about using Linux Containers for isolation of Map/Reduce processes. I don't know if any JIRA has been filed for it or not, but they are much lighter weight then Xen and other virtualization tech, because all it really is concerned with is resource isolation, and not virtualizing an entire operating system.
>
> --Bobby Evans
>
> On 9/9/11 10:58 AM, "Saikat Kanjilal" <sx...@hotmail.com> wrote:
>
>
>
> Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
> Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace
> I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
> Regards
>
Re: Research projects for hadoop
Posted by Robert Evans <ev...@yahoo-inc.com>.
The biggest issue with Xen and other virtualization technologies is that often there is an IO penalty involved with using them. For many jobs this is not an acceptable trade off. I do know, however, that there has been some discussion about using Linux Containers for isolation of Map/Reduce processes. I don't know if any JIRA has been filed for it or not, but they are much lighter weight then Xen and other virtualization tech, because all it really is concerned with is resource isolation, and not virtualizing an entire operating system.
--Bobby Evans
On 9/9/11 10:58 AM, "Saikat Kanjilal" <sx...@hotmail.com> wrote:
Hi Folks,I was looking through the following wiki page: http://wiki.apache.org/hadoop/HadoopResearchProjects and was wondering if there's been any work done (or any interest to do work) for the following topics:
Integration of Virtualization (such as Xen) with Hadoop toolsHow does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor?As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits:By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable.With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters.With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot.Provisioning of long running Services via HODWork on a computation model for services on the grid. The model would include:Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractionsLogical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementationA few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions,Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc.Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc.Integration with HDFS for persistence, as well as access to local filesystemsIntegration with ZooKeeper so that applications can use the namespace
I would like to either help out with a design for the above or prototyping code, please let me know if and what the process may be to move forward with this.
Regards