Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

We are exploring the possibility of hosting Hadoop outside of our data centers. I am aware that Hadoop in general isn't exactly designed to run on virtual hardware. So a few questions:
1. Are there any providers out there who would host Hadoop on dedicated physical hardware? 
2. Has anyone had success hosting Hadoop on virtualized hardware where 100% uptime and performance/stability are very important (we use HBase as a real time database and it needs to be up all the time)?
 
Thanks,
Dhaval

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for sharing the experience Alex. I kind of anticipated the kind of issues you mentioned here but just wanted to make sure I explore all possible options
 
Regards,
Dhaval



On Wednesday, 16 October 2013 1:34 PM, alex bohr <al...@gmail.com> wrote:
 
Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about regarding random resource constraints or random outages that might happen when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU, and more importantly they don't have ideal instance types.  The M1.xlarges are too small storage wise (we ended up paying for more CPU than we needed to get the amount of storage we needed) and the hs1.8xlarge are too big - they have 24 drives and it feels like we lose a good amount of CPU controlling IO across all those drives, and we now have significantly more storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you already have hardware, moving to AWS I think will increase your monthly bills to get comparable performance.



On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
> 
>Regards,
>Dhaval
>
>
>
>________________________________
> From: alex bohr <al...@gmail.com>
>To: user@hadoop.apache.org 
>Cc: Dhaval Shah <pr...@yahoo.co.in> 
>Sent: Monday, 12 August 2013 1:41 PM
>Subject: Re: Hosting Hadoop
> 
>
>
>I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
>I don't think EMR is an ideal solution if your cluster will be running 24/7.
>
>
>But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?
>
>
>
>On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
>check altiscale as well
>>
>>
>>
>>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>>
>>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>>> 
>>>Regards,
>>>Dhaval
>>>
>>>
>>>
>>>________________________________
>>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>>Cc: user@hadoop.apache.org 
>>>Sent: Thursday, 8 August 2013 4:50 PM
>>>Subject: Re: Hosting Hadoop
>>> 
>>>
>>>Well, all depends, because many companies use Cloud Computing
>>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>>hosting:
>>>http://aws.amazon.com/elasticmapreduce
>>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>>http://bitrefinery.com/services/hadoop-hosting
>>>http://www.joyent.com/products/compute-service/features/hadoop
>>>
>>>There a lot of companies using HBase hosted in Cloud. The last
>>>HBaseCon was full of great use-cases:
>>>HBase at
 Pinterest:
>>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>>
>>>HBase at Groupon
>>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>>
>>>A great talk by Benoit for Networking design for HBase:
>>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>>
>>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>>
>>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>>> We are exploring the possibility of hosting Hadoop outside of our data
>>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>>> virtual hardware. So a few questions:
>>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>>> physical hardware?
>>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>>> uptime and performance/stability are very important (we use HBase as a real
>>>> time database and it needs to be up all the time)?
>>>>
>>>> Thanks,
>>>> Dhaval
>>>
>>>
>>>-- 
>>>Marcos Ortiz Valmaseda
>>>Product Manager at PDVSA
>>>http://about.me/marcosortiz
>>>
>>>
>>>
>>
>>
>>
>>-- 
>>Nitin Pawar
>>
>
>
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for sharing the experience Alex. I kind of anticipated the kind of issues you mentioned here but just wanted to make sure I explore all possible options
 
Regards,
Dhaval



On Wednesday, 16 October 2013 1:34 PM, alex bohr <al...@gmail.com> wrote:
 
Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about regarding random resource constraints or random outages that might happen when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU, and more importantly they don't have ideal instance types.  The M1.xlarges are too small storage wise (we ended up paying for more CPU than we needed to get the amount of storage we needed) and the hs1.8xlarge are too big - they have 24 drives and it feels like we lose a good amount of CPU controlling IO across all those drives, and we now have significantly more storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you already have hardware, moving to AWS I think will increase your monthly bills to get comparable performance.



On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
> 
>Regards,
>Dhaval
>
>
>
>________________________________
> From: alex bohr <al...@gmail.com>
>To: user@hadoop.apache.org 
>Cc: Dhaval Shah <pr...@yahoo.co.in> 
>Sent: Monday, 12 August 2013 1:41 PM
>Subject: Re: Hosting Hadoop
> 
>
>
>I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
>I don't think EMR is an ideal solution if your cluster will be running 24/7.
>
>
>But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?
>
>
>
>On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
>check altiscale as well
>>
>>
>>
>>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>>
>>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>>> 
>>>Regards,
>>>Dhaval
>>>
>>>
>>>
>>>________________________________
>>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>>Cc: user@hadoop.apache.org 
>>>Sent: Thursday, 8 August 2013 4:50 PM
>>>Subject: Re: Hosting Hadoop
>>> 
>>>
>>>Well, all depends, because many companies use Cloud Computing
>>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>>hosting:
>>>http://aws.amazon.com/elasticmapreduce
>>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>>http://bitrefinery.com/services/hadoop-hosting
>>>http://www.joyent.com/products/compute-service/features/hadoop
>>>
>>>There a lot of companies using HBase hosted in Cloud. The last
>>>HBaseCon was full of great use-cases:
>>>HBase at
 Pinterest:
>>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>>
>>>HBase at Groupon
>>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>>
>>>A great talk by Benoit for Networking design for HBase:
>>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>>
>>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>>
>>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>>> We are exploring the possibility of hosting Hadoop outside of our data
>>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>>> virtual hardware. So a few questions:
>>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>>> physical hardware?
>>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>>> uptime and performance/stability are very important (we use HBase as a real
>>>> time database and it needs to be up all the time)?
>>>>
>>>> Thanks,
>>>> Dhaval
>>>
>>>
>>>-- 
>>>Marcos Ortiz Valmaseda
>>>Product Manager at PDVSA
>>>http://about.me/marcosortiz
>>>
>>>
>>>
>>
>>
>>
>>-- 
>>Nitin Pawar
>>
>
>
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for sharing the experience Alex. I kind of anticipated the kind of issues you mentioned here but just wanted to make sure I explore all possible options
 
Regards,
Dhaval



On Wednesday, 16 October 2013 1:34 PM, alex bohr <al...@gmail.com> wrote:
 
Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about regarding random resource constraints or random outages that might happen when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU, and more importantly they don't have ideal instance types.  The M1.xlarges are too small storage wise (we ended up paying for more CPU than we needed to get the amount of storage we needed) and the hs1.8xlarge are too big - they have 24 drives and it feels like we lose a good amount of CPU controlling IO across all those drives, and we now have significantly more storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you already have hardware, moving to AWS I think will increase your monthly bills to get comparable performance.



On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
> 
>Regards,
>Dhaval
>
>
>
>________________________________
> From: alex bohr <al...@gmail.com>
>To: user@hadoop.apache.org 
>Cc: Dhaval Shah <pr...@yahoo.co.in> 
>Sent: Monday, 12 August 2013 1:41 PM
>Subject: Re: Hosting Hadoop
> 
>
>
>I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
>I don't think EMR is an ideal solution if your cluster will be running 24/7.
>
>
>But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?
>
>
>
>On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
>check altiscale as well
>>
>>
>>
>>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>>
>>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>>> 
>>>Regards,
>>>Dhaval
>>>
>>>
>>>
>>>________________________________
>>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>>Cc: user@hadoop.apache.org 
>>>Sent: Thursday, 8 August 2013 4:50 PM
>>>Subject: Re: Hosting Hadoop
>>> 
>>>
>>>Well, all depends, because many companies use Cloud Computing
>>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>>hosting:
>>>http://aws.amazon.com/elasticmapreduce
>>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>>http://bitrefinery.com/services/hadoop-hosting
>>>http://www.joyent.com/products/compute-service/features/hadoop
>>>
>>>There a lot of companies using HBase hosted in Cloud. The last
>>>HBaseCon was full of great use-cases:
>>>HBase at
 Pinterest:
>>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>>
>>>HBase at Groupon
>>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>>
>>>A great talk by Benoit for Networking design for HBase:
>>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>>
>>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>>
>>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>>> We are exploring the possibility of hosting Hadoop outside of our data
>>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>>> virtual hardware. So a few questions:
>>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>>> physical hardware?
>>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>>> uptime and performance/stability are very important (we use HBase as a real
>>>> time database and it needs to be up all the time)?
>>>>
>>>> Thanks,
>>>> Dhaval
>>>
>>>
>>>-- 
>>>Marcos Ortiz Valmaseda
>>>Product Manager at PDVSA
>>>http://about.me/marcosortiz
>>>
>>>
>>>
>>
>>
>>
>>-- 
>>Nitin Pawar
>>
>
>
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for sharing the experience Alex. I kind of anticipated the kind of issues you mentioned here but just wanted to make sure I explore all possible options
 
Regards,
Dhaval



On Wednesday, 16 October 2013 1:34 PM, alex bohr <al...@gmail.com> wrote:
 
Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about regarding random resource constraints or random outages that might happen when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU, and more importantly they don't have ideal instance types.  The M1.xlarges are too small storage wise (we ended up paying for more CPU than we needed to get the amount of storage we needed) and the hs1.8xlarge are too big - they have 24 drives and it feels like we lose a good amount of CPU controlling IO across all those drives, and we now have significantly more storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you already have hardware, moving to AWS I think will increase your monthly bills to get comparable performance.



On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
> 
>Regards,
>Dhaval
>
>
>
>________________________________
> From: alex bohr <al...@gmail.com>
>To: user@hadoop.apache.org 
>Cc: Dhaval Shah <pr...@yahoo.co.in> 
>Sent: Monday, 12 August 2013 1:41 PM
>Subject: Re: Hosting Hadoop
> 
>
>
>I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
>I don't think EMR is an ideal solution if your cluster will be running 24/7.
>
>
>But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?
>
>
>
>On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
>check altiscale as well
>>
>>
>>
>>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>>
>>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>>> 
>>>Regards,
>>>Dhaval
>>>
>>>
>>>
>>>________________________________
>>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>>Cc: user@hadoop.apache.org 
>>>Sent: Thursday, 8 August 2013 4:50 PM
>>>Subject: Re: Hosting Hadoop
>>> 
>>>
>>>Well, all depends, because many companies use Cloud Computing
>>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>>hosting:
>>>http://aws.amazon.com/elasticmapreduce
>>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>>http://bitrefinery.com/services/hadoop-hosting
>>>http://www.joyent.com/products/compute-service/features/hadoop
>>>
>>>There a lot of companies using HBase hosted in Cloud. The last
>>>HBaseCon was full of great use-cases:
>>>HBase at
 Pinterest:
>>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>>
>>>HBase at Groupon
>>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>>
>>>A great talk by Benoit for Networking design for HBase:
>>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>>
>>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>>
>>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>>> We are exploring the possibility of hosting Hadoop outside of our data
>>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>>> virtual hardware. So a few questions:
>>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>>> physical hardware?
>>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>>> uptime and performance/stability are very important (we use HBase as a real
>>>> time database and it needs to be up all the time)?
>>>>
>>>> Thanks,
>>>> Dhaval
>>>
>>>
>>>-- 
>>>Marcos Ortiz Valmaseda
>>>Product Manager at PDVSA
>>>http://about.me/marcosortiz
>>>
>>>
>>>
>>
>>
>>
>>-- 
>>Nitin Pawar
>>
>
>
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about
regarding random resource constraints or random outages that might happen
when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU,
and more importantly they don't have ideal instance types.  The M1.xlarges
are too small storage wise (we ended up paying for more CPU than we needed
to get the amount of storage we needed) and the hs1.8xlarge are too big -
they have 24 drives and it feels like we lose a good amount of CPU
controlling IO across all those drives, and we now have significantly more
storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you
already have hardware, moving to AWS I think will increase your monthly
bills to get comparable performance.


On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about
regarding random resource constraints or random outages that might happen
when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU,
and more importantly they don't have ideal instance types.  The M1.xlarges
are too small storage wise (we ended up paying for more CPU than we needed
to get the amount of storage we needed) and the hs1.8xlarge are too big -
they have 24 drives and it feels like we lose a good amount of CPU
controlling IO across all those drives, and we now have significantly more
storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you
already have hardware, moving to AWS I think will increase your monthly
bills to get comparable performance.


On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

Dhaval,

It will also depend on how much data and what capacity cluster you are
building.

I agree that costs and efforts on setting up on aws for the beginning are
very less and it speeds up your developments, but if you are planning a
large cluster then you will need to think over capital costs over minimum 3
years for own hardware over setting up things on aws.

Here is one article,
http://deepvalue.net/ec2-is-380-more-expensive-than-internal-cluster/

PS: I run hadoop clusters on aws currently on aws but they are really small
scale cluster and hardly used so have not run into any issues.


On Thu, Aug 22, 2013 at 12:06 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about
regarding random resource constraints or random outages that might happen
when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU,
and more importantly they don't have ideal instance types.  The M1.xlarges
are too small storage wise (we ended up paying for more CPU than we needed
to get the amount of storage we needed) and the hs1.8xlarge are too big -
they have 24 drives and it feels like we lose a good amount of CPU
controlling IO across all those drives, and we now have significantly more
storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you
already have hardware, moving to AWS I think will increase your monthly
bills to get comparable performance.


On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

Dhaval,

It will also depend on how much data and what capacity cluster you are
building.

I agree that costs and efforts on setting up on aws for the beginning are
very less and it speeds up your developments, but if you are planning a
large cluster then you will need to think over capital costs over minimum 3
years for own hardware over setting up things on aws.

Here is one article,
http://deepvalue.net/ec2-is-380-more-expensive-than-internal-cluster/

PS: I run hadoop clusters on aws currently on aws but they are really small
scale cluster and hardly used so have not run into any issues.


On Thu, Aug 22, 2013 at 12:06 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

Hi Dhaval,
Sorry just saw this email (oops) so might not be relevant - but:
We didn't encounter too much Funky issues that we were worried about
regarding random resource constraints or random outages that might happen
when sharing a physical box with unknown neighbors.

But overall we feel the virtualization is robbing us of significant CPU,
and more importantly they don't have ideal instance types.  The M1.xlarges
are too small storage wise (we ended up paying for more CPU than we needed
to get the amount of storage we needed) and the hs1.8xlarge are too big -
they have 24 drives and it feels like we lose a good amount of CPU
controlling IO across all those drives, and we now have significantly more
storage than we need in order to get enough CPU to keep our SLAs.

For initial set-up - AWS is way quicker than owning hardware.  But if you
already have hardware, moving to AWS I think will increase your monthly
bills to get comparable performance.


On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

Dhaval,

It will also depend on how much data and what capacity cluster you are
building.

I agree that costs and efforts on setting up on aws for the beginning are
very less and it speeds up your developments, but if you are planning a
large cluster then you will need to think over capital costs over minimum 3
years for own hardware over setting up things on aws.

Here is one article,
http://deepvalue.net/ec2-is-380-more-expensive-than-internal-cluster/

PS: I run hadoop clusters on aws currently on aws but they are really small
scale cluster and hardly used so have not run into any issues.


On Thu, Aug 22, 2013 at 12:06 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

Dhaval,

It will also depend on how much data and what capacity cluster you are
building.

I agree that costs and efforts on setting up on aws for the beginning are
very less and it speeds up your developments, but if you are planning a
large cluster then you will need to think over capital costs over minimum 3
years for own hardware over setting up things on aws.

Here is one article,
http://deepvalue.net/ec2-is-380-more-expensive-than-internal-cluster/

PS: I run hadoop clusters on aws currently on aws but they are really small
scale cluster and hardly used so have not run into any issues.


On Thu, Aug 22, 2013 at 12:06 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:

> Alex, did you run into funky issues with EC2/EMR? The kind of issues that
> would come up because its a virtualized environment? We currently own our
> hardware and are just trying to do an ROI analysis on whether moving to
> Amazon can reduce our admin costs. Currently administering a Hadoop cluster
> is a bit expensive (in terms of man hours spent trying to replace disks and
> so on) and we are exploring whether its possible to avoid some of those
> costs
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* alex bohr <al...@gmail.com>
> *To:* user@hadoop.apache.org
> *Cc:* Dhaval Shah <pr...@yahoo.co.in>
> *Sent:* Monday, 12 August 2013 1:41 PM
> *Subject:* Re: Hosting Hadoop
>
> I've had good experience running a large hadoop cluster on EC2 instances.
>  After almost 1 year we haven't had any significant down time, just lost a
> small # of data nodes.
> I don't think EMR is an ideal solution if your cluster will be running
> 24/7.
>
> But for running a large cluster, I don't see how you it's more cost
> efficient to run in the cloud than to own the hardware and we're trying to
> move off the cloud onto our own hardware.  Can I ask why you're looking to
> move to the cloud?
>
>
> On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>
>
>
>  --
> Nitin Pawar
>
>
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
 
Regards,
Dhaval


________________________________
 From: alex bohr <al...@gmail.com>
To: user@hadoop.apache.org 
Cc: Dhaval Shah <pr...@yahoo.co.in> 
Sent: Monday, 12 August 2013 1:41 PM
Subject: Re: Hosting Hadoop
 


I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?



On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:

check altiscale as well
>
>
>
>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>
>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>> 
>>Regards,
>>Dhaval
>>
>>
>>
>>________________________________
>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>Cc: user@hadoop.apache.org 
>>Sent: Thursday, 8 August 2013 4:50 PM
>>Subject: Re: Hosting Hadoop
>> 
>>
>>Well, all depends, because many companies use Cloud Computing
>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>hosting:
>>http://aws.amazon.com/elasticmapreduce
>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>http://bitrefinery.com/services/hadoop-hosting
>>http://www.joyent.com/products/compute-service/features/hadoop
>>
>>There a lot of companies using HBase hosted in Cloud. The last
>>HBaseCon was full of great use-cases:
>>HBase at
 Pinterest:
>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>>HBase at Groupon
>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>>A great talk by Benoit for Networking design for HBase:
>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>> We are exploring the possibility of hosting Hadoop outside of our data
>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>> virtual hardware. So a few questions:
>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>> physical hardware?
>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>> uptime and performance/stability are very important (we use HBase as a real
>>> time database and it needs to be up all the time)?
>>>
>>> Thanks,
>>> Dhaval
>>
>>
>>-- 
>>Marcos Ortiz Valmaseda
>>Product Manager at PDVSA
>>http://about.me/marcosortiz
>>
>>
>>
>
>
>
>-- 
>Nitin Pawar
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
 
Regards,
Dhaval


________________________________
 From: alex bohr <al...@gmail.com>
To: user@hadoop.apache.org 
Cc: Dhaval Shah <pr...@yahoo.co.in> 
Sent: Monday, 12 August 2013 1:41 PM
Subject: Re: Hosting Hadoop
 


I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?



On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:

check altiscale as well
>
>
>
>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>
>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>> 
>>Regards,
>>Dhaval
>>
>>
>>
>>________________________________
>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>Cc: user@hadoop.apache.org 
>>Sent: Thursday, 8 August 2013 4:50 PM
>>Subject: Re: Hosting Hadoop
>> 
>>
>>Well, all depends, because many companies use Cloud Computing
>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>hosting:
>>http://aws.amazon.com/elasticmapreduce
>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>http://bitrefinery.com/services/hadoop-hosting
>>http://www.joyent.com/products/compute-service/features/hadoop
>>
>>There a lot of companies using HBase hosted in Cloud. The last
>>HBaseCon was full of great use-cases:
>>HBase at
 Pinterest:
>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>>HBase at Groupon
>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>>A great talk by Benoit for Networking design for HBase:
>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>> We are exploring the possibility of hosting Hadoop outside of our data
>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>> virtual hardware. So a few questions:
>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>> physical hardware?
>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>> uptime and performance/stability are very important (we use HBase as a real
>>> time database and it needs to be up all the time)?
>>>
>>> Thanks,
>>> Dhaval
>>
>>
>>-- 
>>Marcos Ortiz Valmaseda
>>Product Manager at PDVSA
>>http://about.me/marcosortiz
>>
>>
>>
>
>
>
>-- 
>Nitin Pawar
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
 
Regards,
Dhaval


________________________________
 From: alex bohr <al...@gmail.com>
To: user@hadoop.apache.org 
Cc: Dhaval Shah <pr...@yahoo.co.in> 
Sent: Monday, 12 August 2013 1:41 PM
Subject: Re: Hosting Hadoop
 


I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?



On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:

check altiscale as well
>
>
>
>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>
>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>> 
>>Regards,
>>Dhaval
>>
>>
>>
>>________________________________
>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>Cc: user@hadoop.apache.org 
>>Sent: Thursday, 8 August 2013 4:50 PM
>>Subject: Re: Hosting Hadoop
>> 
>>
>>Well, all depends, because many companies use Cloud Computing
>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>hosting:
>>http://aws.amazon.com/elasticmapreduce
>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>http://bitrefinery.com/services/hadoop-hosting
>>http://www.joyent.com/products/compute-service/features/hadoop
>>
>>There a lot of companies using HBase hosted in Cloud. The last
>>HBaseCon was full of great use-cases:
>>HBase at
 Pinterest:
>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>>HBase at Groupon
>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>>A great talk by Benoit for Networking design for HBase:
>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>> We are exploring the possibility of hosting Hadoop outside of our data
>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>> virtual hardware. So a few questions:
>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>> physical hardware?
>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>> uptime and performance/stability are very important (we use HBase as a real
>>> time database and it needs to be up all the time)?
>>>
>>> Thanks,
>>> Dhaval
>>
>>
>>-- 
>>Marcos Ortiz Valmaseda
>>Product Manager at PDVSA
>>http://about.me/marcosortiz
>>
>>
>>
>
>
>
>-- 
>Nitin Pawar
>

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Alex, did you run into funky issues with EC2/EMR? The kind of issues that would come up because its a virtualized environment? We currently own our hardware and are just trying to do an ROI analysis on whether moving to Amazon can reduce our admin costs. Currently administering a Hadoop cluster is a bit expensive (in terms of man hours spent trying to replace disks and so on) and we are exploring whether its possible to avoid some of those costs
 
Regards,
Dhaval


________________________________
 From: alex bohr <al...@gmail.com>
To: user@hadoop.apache.org 
Cc: Dhaval Shah <pr...@yahoo.co.in> 
Sent: Monday, 12 August 2013 1:41 PM
Subject: Re: Hosting Hadoop
 


I've had good experience running a large hadoop cluster on EC2 instances.  After almost 1 year we haven't had any significant down time, just lost a small # of data nodes.  
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost efficient to run in the cloud than to own the hardware and we're trying to move off the cloud onto our own hardware.  Can I ask why you're looking to move to the cloud?



On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com> wrote:

check altiscale as well
>
>
>
>On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in> wrote:
>
>Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
>> 
>>Regards,
>>Dhaval
>>
>>
>>
>>________________________________
>> From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>>To: Dhaval Shah <pr...@yahoo.co.in> 
>>Cc: user@hadoop.apache.org 
>>Sent: Thursday, 8 August 2013 4:50 PM
>>Subject: Re: Hosting Hadoop
>> 
>>
>>Well, all depends, because many companies use Cloud Computing
>>platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>>hosting:
>>http://aws.amazon.com/elasticmapreduce
>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>>http://bitrefinery.com/services/hadoop-hosting
>>http://www.joyent.com/products/compute-service/features/hadoop
>>
>>There a lot of companies using HBase hosted in Cloud. The last
>>HBaseCon was full of great use-cases:
>>HBase at
 Pinterest:
>>http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>>HBase at Groupon
>>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>>A great talk by Benoit for Networking design for HBase:
>>http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>>Using Coprocessors to Index Columns in an Elasticsearch Cluster
>>http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>>2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>>> We are exploring the possibility of hosting Hadoop outside of our data
>>> centers. I am aware that Hadoop in general isn't exactly designed to run on
>>> virtual hardware. So a few questions:
>>> 1. Are there any providers out there who would host Hadoop on dedicated
>>> physical hardware?
>>> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
>>> uptime and performance/stability are very important (we use HBase as a real
>>> time database and it needs to be up all the time)?
>>>
>>> Thanks,
>>> Dhaval
>>
>>
>>-- 
>>Marcos Ortiz Valmaseda
>>Product Manager at PDVSA
>>http://about.me/marcosortiz
>>
>>
>>
>
>
>
>-- 
>Nitin Pawar
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

I've had good experience running a large hadoop cluster on EC2 instances.
 After almost 1 year we haven't had any significant down time, just lost a
small # of data nodes.
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost
efficient to run in the cloud than to own the hardware and we're trying to
move off the cloud onto our own hardware.  Can I ask why you're looking to
move to the cloud?


On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:

> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
>> Thanks for the list Marcos. I will go through the slides/links. I think
>> that's helpful
>>
>> Regards,
>> Dhaval
>>
>>   ------------------------------
>>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>> *To:* Dhaval Shah <pr...@yahoo.co.in>
>> *Cc:* user@hadoop.apache.org
>> *Sent:* Thursday, 8 August 2013 4:50 PM
>> *Subject:* Re: Hosting Hadoop
>>
>> Well, all depends, because many companies use Cloud Computing
>> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>> hosting:
>> http://aws.amazon.com/elasticmapreduce
>> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>> http://bitrefinery.com/services/hadoop-hosting
>> http://www.joyent.com/products/compute-service/features/hadoop
>>
>> There a lot of companies using HBase hosted in Cloud. The last
>> HBaseCon was full of great use-cases:
>> HBase at Pinterest:
>> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>> HBase at Groupon
>> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>> A great talk by Benoit for Networking design for HBase:
>>
>> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>> Using Coprocessors to Index Columns in an Elasticsearch Cluster
>> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>> > We are exploring the possibility of hosting Hadoop outside of our data
>> > centers. I am aware that Hadoop in general isn't exactly designed to
>> run on
>> > virtual hardware. So a few questions:
>> > 1. Are there any providers out there who would host Hadoop on dedicated
>> > physical hardware?
>> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
>> 100%
>> > uptime and performance/stability are very important (we use HBase as a
>> real
>> > time database and it needs to be up all the time)?
>> >
>> > Thanks,
>> > Dhaval
>>
>>
>> --
>> Marcos Ortiz Valmaseda
>> Product Manager at PDVSA
>> http://about.me/marcosortiz
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

I've had good experience running a large hadoop cluster on EC2 instances.
 After almost 1 year we haven't had any significant down time, just lost a
small # of data nodes.
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost
efficient to run in the cloud than to own the hardware and we're trying to
move off the cloud onto our own hardware.  Can I ask why you're looking to
move to the cloud?


On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:

> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
>> Thanks for the list Marcos. I will go through the slides/links. I think
>> that's helpful
>>
>> Regards,
>> Dhaval
>>
>>   ------------------------------
>>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>> *To:* Dhaval Shah <pr...@yahoo.co.in>
>> *Cc:* user@hadoop.apache.org
>> *Sent:* Thursday, 8 August 2013 4:50 PM
>> *Subject:* Re: Hosting Hadoop
>>
>> Well, all depends, because many companies use Cloud Computing
>> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>> hosting:
>> http://aws.amazon.com/elasticmapreduce
>> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>> http://bitrefinery.com/services/hadoop-hosting
>> http://www.joyent.com/products/compute-service/features/hadoop
>>
>> There a lot of companies using HBase hosted in Cloud. The last
>> HBaseCon was full of great use-cases:
>> HBase at Pinterest:
>> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>> HBase at Groupon
>> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>> A great talk by Benoit for Networking design for HBase:
>>
>> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>> Using Coprocessors to Index Columns in an Elasticsearch Cluster
>> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>> > We are exploring the possibility of hosting Hadoop outside of our data
>> > centers. I am aware that Hadoop in general isn't exactly designed to
>> run on
>> > virtual hardware. So a few questions:
>> > 1. Are there any providers out there who would host Hadoop on dedicated
>> > physical hardware?
>> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
>> 100%
>> > uptime and performance/stability are very important (we use HBase as a
>> real
>> > time database and it needs to be up all the time)?
>> >
>> > Thanks,
>> > Dhaval
>>
>>
>> --
>> Marcos Ortiz Valmaseda
>> Product Manager at PDVSA
>> http://about.me/marcosortiz
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

I've had good experience running a large hadoop cluster on EC2 instances.
 After almost 1 year we haven't had any significant down time, just lost a
small # of data nodes.
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost
efficient to run in the cloud than to own the hardware and we're trying to
move off the cloud onto our own hardware.  Can I ask why you're looking to
move to the cloud?


On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:

> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
>> Thanks for the list Marcos. I will go through the slides/links. I think
>> that's helpful
>>
>> Regards,
>> Dhaval
>>
>>   ------------------------------
>>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>> *To:* Dhaval Shah <pr...@yahoo.co.in>
>> *Cc:* user@hadoop.apache.org
>> *Sent:* Thursday, 8 August 2013 4:50 PM
>> *Subject:* Re: Hosting Hadoop
>>
>> Well, all depends, because many companies use Cloud Computing
>> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>> hosting:
>> http://aws.amazon.com/elasticmapreduce
>> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>> http://bitrefinery.com/services/hadoop-hosting
>> http://www.joyent.com/products/compute-service/features/hadoop
>>
>> There a lot of companies using HBase hosted in Cloud. The last
>> HBaseCon was full of great use-cases:
>> HBase at Pinterest:
>> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>> HBase at Groupon
>> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>> A great talk by Benoit for Networking design for HBase:
>>
>> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>> Using Coprocessors to Index Columns in an Elasticsearch Cluster
>> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>> > We are exploring the possibility of hosting Hadoop outside of our data
>> > centers. I am aware that Hadoop in general isn't exactly designed to
>> run on
>> > virtual hardware. So a few questions:
>> > 1. Are there any providers out there who would host Hadoop on dedicated
>> > physical hardware?
>> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
>> 100%
>> > uptime and performance/stability are very important (we use HBase as a
>> real
>> > time database and it needs to be up all the time)?
>> >
>> > Thanks,
>> > Dhaval
>>
>>
>> --
>> Marcos Ortiz Valmaseda
>> Product Manager at PDVSA
>> http://about.me/marcosortiz
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hosting Hadoop

[alex bohr <al...@gmail.com>]

I've had good experience running a large hadoop cluster on EC2 instances.
 After almost 1 year we haven't had any significant down time, just lost a
small # of data nodes.
I don't think EMR is an ideal solution if your cluster will be running 24/7.

But for running a large cluster, I don't see how you it's more cost
efficient to run in the cloud than to own the hardware and we're trying to
move off the cloud onto our own hardware.  Can I ask why you're looking to
move to the cloud?


On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <ni...@gmail.com>wrote:

> check altiscale as well
>
>
> On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:
>
>> Thanks for the list Marcos. I will go through the slides/links. I think
>> that's helpful
>>
>> Regards,
>> Dhaval
>>
>>   ------------------------------
>>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
>> *To:* Dhaval Shah <pr...@yahoo.co.in>
>> *Cc:* user@hadoop.apache.org
>> *Sent:* Thursday, 8 August 2013 4:50 PM
>> *Subject:* Re: Hosting Hadoop
>>
>> Well, all depends, because many companies use Cloud Computing
>> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
>> hosting:
>> http://aws.amazon.com/elasticmapreduce
>> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
>> http://bitrefinery.com/services/hadoop-hosting
>> http://www.joyent.com/products/compute-service/features/hadoop
>>
>> There a lot of companies using HBase hosted in Cloud. The last
>> HBaseCon was full of great use-cases:
>> HBase at Pinterest:
>> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>>
>> HBase at Groupon
>> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>>
>> A great talk by Benoit for Networking design for HBase:
>>
>> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>>
>> Using Coprocessors to Index Columns in an Elasticsearch Cluster
>> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>>
>> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
>> > We are exploring the possibility of hosting Hadoop outside of our data
>> > centers. I am aware that Hadoop in general isn't exactly designed to
>> run on
>> > virtual hardware. So a few questions:
>> > 1. Are there any providers out there who would host Hadoop on dedicated
>> > physical hardware?
>> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
>> 100%
>> > uptime and performance/stability are very important (we use HBase as a
>> real
>> > time database and it needs to be up all the time)?
>> >
>> > Thanks,
>> > Dhaval
>>
>>
>> --
>> Marcos Ortiz Valmaseda
>> Product Manager at PDVSA
>> http://about.me/marcosortiz
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

check altiscale as well


On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:

> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

check altiscale as well


On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:

> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

check altiscale as well


On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:

> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Nitin Pawar <ni...@gmail.com>]

check altiscale as well


On Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <pr...@yahoo.co.in>wrote:

> Thanks for the list Marcos. I will go through the slides/links. I think
> that's helpful
>
> Regards,
> Dhaval
>
>   ------------------------------
>  *From:* Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> *To:* Dhaval Shah <pr...@yahoo.co.in>
> *Cc:* user@hadoop.apache.org
> *Sent:* Thursday, 8 August 2013 4:50 PM
> *Subject:* Re: Hosting Hadoop
>
> Well, all depends, because many companies use Cloud Computing
> platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
> hosting:
> http://aws.amazon.com/elasticmapreduce
> http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
> http://bitrefinery.com/services/hadoop-hosting
> http://www.joyent.com/products/compute-service/features/hadoop
>
> There a lot of companies using HBase hosted in Cloud. The last
> HBaseCon was full of great use-cases:
> HBase at Pinterest:
> http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
>
> HBase at Groupon
> http://www.hbasecon.com/sessions/apache-hbase-at-groupon/
>
> A great talk by Benoit for Networking design for HBase:
> http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/
>
> Using Coprocessors to Index Columns in an Elasticsearch Cluster
> http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/
>
> 2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> > We are exploring the possibility of hosting Hadoop outside of our data
> > centers. I am aware that Hadoop in general isn't exactly designed to run
> on
> > virtual hardware. So a few questions:
> > 1. Are there any providers out there who would host Hadoop on dedicated
> > physical hardware?
> > 2. Has anyone had success hosting Hadoop on virtualized hardware where
> 100%
> > uptime and performance/stability are very important (we use HBase as a
> real
> > time database and it needs to be up all the time)?
> >
> > Thanks,
> > Dhaval
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
>


-- 
Nitin Pawar

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
 
Regards,
Dhaval


________________________________
 From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
To: Dhaval Shah <pr...@yahoo.co.in> 
Cc: user@hadoop.apache.org 
Sent: Thursday, 8 August 2013 4:50 PM
Subject: Re: Hosting Hadoop
 

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
 
Regards,
Dhaval


________________________________
 From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
To: Dhaval Shah <pr...@yahoo.co.in> 
Cc: user@hadoop.apache.org 
Sent: Thursday, 8 August 2013 4:50 PM
Subject: Re: Hosting Hadoop
 

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
 
Regards,
Dhaval


________________________________
 From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
To: Dhaval Shah <pr...@yahoo.co.in> 
Cc: user@hadoop.apache.org 
Sent: Thursday, 8 August 2013 4:50 PM
Subject: Re: Hosting Hadoop
 

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Dhaval Shah <pr...@yahoo.co.in>]

Thanks for the list Marcos. I will go through the slides/links. I think that's helpful
 
Regards,
Dhaval


________________________________
 From: Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
To: Dhaval Shah <pr...@yahoo.co.in> 
Cc: user@hadoop.apache.org 
Sent: Thursday, 8 August 2013 4:50 PM
Subject: Re: Hosting Hadoop
 

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Marcos Luis Ortiz Valmaseda <ma...@gmail.com>]

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Marcos Luis Ortiz Valmaseda <ma...@gmail.com>]

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Marcos Luis Ortiz Valmaseda <ma...@gmail.com>]

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz

Re: Hosting Hadoop

[Marcos Luis Ortiz Valmaseda <ma...@gmail.com>]

Well, all depends, because many companies use Cloud Computing
platforms like Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosting:
http://aws.amazon.com/elasticmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.html
http://bitrefinery.com/services/hadoop-hosting
http://www.joyent.com/products/compute-service/features/hadoop

There a lot of companies using HBase hosted in Cloud. The last
HBaseCon was full of great use-cases:
HBase at Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/

HBase at Groupon
http://www.hbasecon.com/sessions/apache-hbase-at-groupon/

A great talk by Benoit for Networking design for HBase:
http://www.hbasecon.com/sessions/scalable-network-designs-for-apache-hbase/

Using Coprocessors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com/sessions/using-coprocessors-to-index-columns/

2013/8/8, Dhaval Shah <pr...@yahoo.co.in>:
> We are exploring the possibility of hosting Hadoop outside of our data
> centers. I am aware that Hadoop in general isn't exactly designed to run on
> virtual hardware. So a few questions:
> 1. Are there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had success hosting Hadoop on virtualized hardware where 100%
> uptime and performance/stability are very important (we use HBase as a real
> time database and it needs to be up all the time)?
>
> Thanks,
> Dhaval


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz