Re: Large feature development (YARN vs HDFS)

[Eric Baldeschwieler <er...@hortonworks.com>]

Referring back to Chris M.s thread, this YARN vs HDFS discussion sounds a lot like an umbrella project issue to me.

On Sep 2, 2012, at 3:11 PM, Arun Murthy wrote:

> Eli,
> 
> On Sep 2, 2012, at 1:01 PM, Eli Collins <el...@cloudera.com> wrote:
> 
>> On Sat, Sep 1, 2012 at 12:47 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>>> Todd,
>>> 
>>> On Sep 1, 2012, at 1:20 AM, Todd Lipcon wrote:
>>> 
>>>> I'd actually contend that YARN was merged too early. I have yet to see
>>>> anyone running YARN in production, and it's holding up the "Stable"
>>>> moniker for Hadoop 2.0 -- HDFS-wise we are already quite stable and
>>>> I'm seeing fewer issues in our customers running Hadoop HDFS 2
>>>> compared to Hadoop 1-derived code.
>>> 
>>> You know I respect you a ton, but I'm very saddened to see you perpetuate this FUD on our public lists. I expected better, particularly when everyone is working towards the same goals of advancing Hadoop-2. This sniping on other members doing work is, um, I'll just stop here rather than regret later.
>> 2. HDFS is more mature than YARN. Not a surprise given that we all
>> agree YARN is alpha, and a much newer project than HDFS that hasn't
>> yet been deployed in production environments yet (to my knowledge).
> 
> Let's focus on the ground reality here.
> 
> Please read my (or Rajiv's) message again about YARN's current
> stability and how much it's baked, it's deployment plans to a very
> large cluster in a few *days*. Or, talk to the people developing,
> testing and supporting these customers and clusters.
> 
> I'll repeat - YARN has clearly baked much more than HDFS HA given
> the basic bugs (upgrade, edit logs corruption etc.) we've seen after
> being declared *done*; but then we just disagree since clearly I'm
> more conservative. Also, we need to be more conservative wrt HDFS -
> but then what would I know...
> 
> I'll admit it's hard to discuss with someone (or a collective) who
> just repeat themselves. Plus, I broke my own rule about email this
> weekend - so, I'll try harder.
> 
> Arun

Re: Large feature development (YARN vs HDFS)

[Arun C Murthy <ac...@hortonworks.com>]

Agreed... it does seem like a case of 'my wife is prettier'.

Maybe I'm oversensitive and it may even be understandable given how much of my waking time I've devoted to YARN over the last 30 months; but I do apologize for indulging in the behavior I accused others of. A good night's sleep does help in clearing mists. IAC, the point I was trying to quantify is simple - current state of YARN is far better than was being characterized here.

We should get back to discussing 'large-feature development' - thanks for starting that discussion Steve.

Arun

On Sep 3, 2012, at 2:30 PM, Eric Baldeschwieler wrote:

> 
> Referring back to Chris M.s thread, this YARN vs HDFS discussion sounds a lot like an umbrella project issue to me.
> 
> On Sep 2, 2012, at 3:11 PM, Arun Murthy wrote:
> 
>> Eli,
>> 
>> On Sep 2, 2012, at 1:01 PM, Eli Collins <el...@cloudera.com> wrote:
>> 
>>> On Sat, Sep 1, 2012 at 12:47 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>>>> Todd,
>>>> 
>>>> On Sep 1, 2012, at 1:20 AM, Todd Lipcon wrote:
>>>> 
>>>>> I'd actually contend that YARN was merged too early. I have yet to see
>>>>> anyone running YARN in production, and it's holding up the "Stable"
>>>>> moniker for Hadoop 2.0 -- HDFS-wise we are already quite stable and
>>>>> I'm seeing fewer issues in our customers running Hadoop HDFS 2
>>>>> compared to Hadoop 1-derived code.
>>>> 
>>>> You know I respect you a ton, but I'm very saddened to see you perpetuate this FUD on our public lists. I expected better, particularly when everyone is working towards the same goals of advancing Hadoop-2. This sniping on other members doing work is, um, I'll just stop here rather than regret later.
>>> 2. HDFS is more mature than YARN. Not a surprise given that we all
>>> agree YARN is alpha, and a much newer project than HDFS that hasn't
>>> yet been deployed in production environments yet (to my knowledge).
>> 
>> Let's focus on the ground reality here.
>> 
>> Please read my (or Rajiv's) message again about YARN's current
>> stability and how much it's baked, it's deployment plans to a very
>> large cluster in a few *days*. Or, talk to the people developing,
>> testing and supporting these customers and clusters.
>> 
>> I'll repeat - YARN has clearly baked much more than HDFS HA given
>> the basic bugs (upgrade, edit logs corruption etc.) we've seen after
>> being declared *done*; but then we just disagree since clearly I'm
>> more conservative. Also, we need to be more conservative wrt HDFS -
>> but then what would I know...
>> 
>> I'll admit it's hard to discuss with someone (or a collective) who
>> just repeat themselves. Plus, I broke my own rule about email this
>> weekend - so, I'll try harder.
>> 
>> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/