You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Matt K <ma...@gmail.com> on 2014/04/24 22:17:54 UTC
running YARN in Production
We run a number of mission-critical MapReduce jobs daily in our production
cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop
bugs, and found it difficult to maintain a solid SLA.
We are now moving to CDH5 and evaluating if we should move to YARN or keep
running Hadoop 1. YARN is very compelling, but it's also relatively young.
I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use
a second opinion :)
Can someone running YARN in a mission-critical Production environment share
their experience, specifically as it relates to stability? I realize that
this is a question that lends itself to a somewhat subjective answer.
Thanks,
-Matt
Re: running YARN in Production
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Matt,
Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too.
For vendor specific questions contact your distribution vendor.
- Alex
Sent from my iPad
> On 24 Apr 2014, at 22:17, Matt K <ma...@gmail.com> wrote:
>
> We run a number of mission-critical MapReduce jobs daily in our production cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep running Hadoop 1. YARN is very compelling, but it's also relatively young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment share their experience, specifically as it relates to stability? I realize that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
Re: running YARN in Production
Posted by Matt K <ma...@gmail.com>.
Allow me to remove "Cloudera" and CDH5 from the question then, and replace
it with "Hadoop 2.3.0".
On Thu, Apr 24, 2014 at 5:49 PM, Marco Shaw <ma...@gmail.com> wrote:
> A bit too many mentions of "Cloudera" for this list... Please consider
> going to a Cloudera list and asking this there asking for specific examples.
>
> Marco
>
>
> On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
>
>> We run a number of mission-critical MapReduce jobs daily in our
>> production cluster, mostly on top of HBase. In the past, we've hit a number
>> of Hadoop bugs, and found it difficult to maintain a solid SLA.
>>
>> We are now moving to CDH5 and evaluating if we should move to YARN or
>> keep running Hadoop 1. YARN is very compelling, but it's also relatively
>> young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I
>> could use a second opinion :)
>>
>> Can someone running YARN in a mission-critical Production environment
>> share their experience, specifically as it relates to stability? I realize
>> that this is a question that lends itself to a somewhat subjective answer.
>>
>> Thanks,
>> -Matt
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: running YARN in Production
Posted by Matt K <ma...@gmail.com>.
Allow me to remove "Cloudera" and CDH5 from the question then, and replace
it with "Hadoop 2.3.0".
On Thu, Apr 24, 2014 at 5:49 PM, Marco Shaw <ma...@gmail.com> wrote:
> A bit too many mentions of "Cloudera" for this list... Please consider
> going to a Cloudera list and asking this there asking for specific examples.
>
> Marco
>
>
> On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
>
>> We run a number of mission-critical MapReduce jobs daily in our
>> production cluster, mostly on top of HBase. In the past, we've hit a number
>> of Hadoop bugs, and found it difficult to maintain a solid SLA.
>>
>> We are now moving to CDH5 and evaluating if we should move to YARN or
>> keep running Hadoop 1. YARN is very compelling, but it's also relatively
>> young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I
>> could use a second opinion :)
>>
>> Can someone running YARN in a mission-critical Production environment
>> share their experience, specifically as it relates to stability? I realize
>> that this is a question that lends itself to a somewhat subjective answer.
>>
>> Thanks,
>> -Matt
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: running YARN in Production
Posted by Matt K <ma...@gmail.com>.
Allow me to remove "Cloudera" and CDH5 from the question then, and replace
it with "Hadoop 2.3.0".
On Thu, Apr 24, 2014 at 5:49 PM, Marco Shaw <ma...@gmail.com> wrote:
> A bit too many mentions of "Cloudera" for this list... Please consider
> going to a Cloudera list and asking this there asking for specific examples.
>
> Marco
>
>
> On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
>
>> We run a number of mission-critical MapReduce jobs daily in our
>> production cluster, mostly on top of HBase. In the past, we've hit a number
>> of Hadoop bugs, and found it difficult to maintain a solid SLA.
>>
>> We are now moving to CDH5 and evaluating if we should move to YARN or
>> keep running Hadoop 1. YARN is very compelling, but it's also relatively
>> young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I
>> could use a second opinion :)
>>
>> Can someone running YARN in a mission-critical Production environment
>> share their experience, specifically as it relates to stability? I realize
>> that this is a question that lends itself to a somewhat subjective answer.
>>
>> Thanks,
>> -Matt
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: running YARN in Production
Posted by Matt K <ma...@gmail.com>.
Allow me to remove "Cloudera" and CDH5 from the question then, and replace
it with "Hadoop 2.3.0".
On Thu, Apr 24, 2014 at 5:49 PM, Marco Shaw <ma...@gmail.com> wrote:
> A bit too many mentions of "Cloudera" for this list... Please consider
> going to a Cloudera list and asking this there asking for specific examples.
>
> Marco
>
>
> On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
>
>> We run a number of mission-critical MapReduce jobs daily in our
>> production cluster, mostly on top of HBase. In the past, we've hit a number
>> of Hadoop bugs, and found it difficult to maintain a solid SLA.
>>
>> We are now moving to CDH5 and evaluating if we should move to YARN or
>> keep running Hadoop 1. YARN is very compelling, but it's also relatively
>> young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I
>> could use a second opinion :)
>>
>> Can someone running YARN in a mission-critical Production environment
>> share their experience, specifically as it relates to stability? I realize
>> that this is a question that lends itself to a somewhat subjective answer.
>>
>> Thanks,
>> -Matt
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: running YARN in Production
Posted by Marco Shaw <ma...@gmail.com>.
A bit too many mentions of "Cloudera" for this list... Please consider
going to a Cloudera list and asking this there asking for specific examples.
Marco
On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
> We run a number of mission-critical MapReduce jobs daily in our production
> cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop
> bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep
> running Hadoop 1. YARN is very compelling, but it's also relatively young.
> I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use
> a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment
> share their experience, specifically as it relates to stability? I realize
> that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
>
Re: running YARN in Production
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Matt,
Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too.
For vendor specific questions contact your distribution vendor.
- Alex
Sent from my iPad
> On 24 Apr 2014, at 22:17, Matt K <ma...@gmail.com> wrote:
>
> We run a number of mission-critical MapReduce jobs daily in our production cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep running Hadoop 1. YARN is very compelling, but it's also relatively young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment share their experience, specifically as it relates to stability? I realize that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
Re: running YARN in Production
Posted by Marco Shaw <ma...@gmail.com>.
A bit too many mentions of "Cloudera" for this list... Please consider
going to a Cloudera list and asking this there asking for specific examples.
Marco
On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
> We run a number of mission-critical MapReduce jobs daily in our production
> cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop
> bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep
> running Hadoop 1. YARN is very compelling, but it's also relatively young.
> I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use
> a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment
> share their experience, specifically as it relates to stability? I realize
> that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
>
Re: running YARN in Production
Posted by Marco Shaw <ma...@gmail.com>.
A bit too many mentions of "Cloudera" for this list... Please consider
going to a Cloudera list and asking this there asking for specific examples.
Marco
On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
> We run a number of mission-critical MapReduce jobs daily in our production
> cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop
> bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep
> running Hadoop 1. YARN is very compelling, but it's also relatively young.
> I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use
> a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment
> share their experience, specifically as it relates to stability? I realize
> that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
>
Re: running YARN in Production
Posted by Marco Shaw <ma...@gmail.com>.
A bit too many mentions of "Cloudera" for this list... Please consider
going to a Cloudera list and asking this there asking for specific examples.
Marco
On Thu, Apr 24, 2014 at 5:17 PM, Matt K <ma...@gmail.com> wrote:
> We run a number of mission-critical MapReduce jobs daily in our production
> cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop
> bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep
> running Hadoop 1. YARN is very compelling, but it's also relatively young.
> I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use
> a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment
> share their experience, specifically as it relates to stability? I realize
> that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
>
Re: running YARN in Production
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Matt,
Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too.
For vendor specific questions contact your distribution vendor.
- Alex
Sent from my iPad
> On 24 Apr 2014, at 22:17, Matt K <ma...@gmail.com> wrote:
>
> We run a number of mission-critical MapReduce jobs daily in our production cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep running Hadoop 1. YARN is very compelling, but it's also relatively young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment share their experience, specifically as it relates to stability? I realize that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt
Re: running YARN in Production
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Matt,
Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too.
For vendor specific questions contact your distribution vendor.
- Alex
Sent from my iPad
> On 24 Apr 2014, at 22:17, Matt K <ma...@gmail.com> wrote:
>
> We run a number of mission-critical MapReduce jobs daily in our production cluster, mostly on top of HBase. In the past, we've hit a number of Hadoop bugs, and found it difficult to maintain a solid SLA.
>
> We are now moving to CDH5 and evaluating if we should move to YARN or keep running Hadoop 1. YARN is very compelling, but it's also relatively young. I know that Cloudera recommends YARN over Hadoop 1 in CDH5, but I could use a second opinion :)
>
> Can someone running YARN in a mission-critical Production environment share their experience, specifically as it relates to stability? I realize that this is a question that lends itself to a somewhat subjective answer.
>
> Thanks,
> -Matt