You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Eric Baldeschwieler <er...@yahoo-inc.com> on 2011/06/17 09:17:56 UTC
Thinking about the next hadoop mainline release
Hi Folks,
I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
- HDFS Federation [HDFS-1052]
- Already merged into trunk
- Next Generation Map-Reduce [MR-279]
- Passing most tests now and discussing merging into trunk
- The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
- This is mostly done, but owen and others are doing a scrub to close out the remaining issues
All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
- New Write Pipeline - HBase support [also in 0.21 & 0.22]
- Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
- Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
Thanks,
E14
Re: Thinking about the next hadoop mainline release
Posted by Steve Loughran <st...@apache.org>.
On 17/06/2011 21:27, Allen Wittenauer wrote:
>
> On Jun 17, 2011, at 10:31 AM, Allen Wittenauer wrote:
>
>>
>> On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security).
>>
>>
>> I'd consider 0.20.203 pseudo-stable.
>
> Actually, I was just reminded about the complete disaster that is metrics. So while it may be pseudo-stable, it isn't actually usable for anyone but Yahoo!.
ooh, I have work on that that I should catch up with and get in.
I'd also like to put in the IBM JVM patch, not out of personal need, but
because I don't like Oracle-JVM dependencies, and it'd be nice to get
IBM's donation into the source tree.
Re: Thinking about the next hadoop mainline release
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Allen,
Can we pls focus on the next release here, on this thread?
Please provide your valuable f/b to the RM abt stuff you'd like to see in the next release.
thanks,
Arun
Sent from my iPhone
On Jun 18, 2011, at 7:45 AM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
> On Jun 17, 2011, at 7:02 PM, Rajiv Chittajallu wrote:
>
>> Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>>>
>>> Actually, I was just reminded about the complete disaster that is metrics. So while it may be pseudo-stable, it isn't actually usable for anyone but Yahoo!.
>>
>> Did you try to use the new metrics framework? Are you complaining
>> cause there is no port of ganglia metrics module to metrics V2?
>
> Like 80-90%+ of the Hadoop sites that gather metrics, yes, we use Ganglia.
>
>> Its pluggable and you can choose to disable it if you want to.
>> All metrics and status information is available via jmx.
>> Multiple sinks and filters, more importantly you can refresh
>> metrics configs without restarting the process.
>
> But first we need to switch to a new metrics infrastructure. This wouldn't have been so bad if the release notes actually had said "You know your current metrics system? Yeah, toss it because we just broke everything."
>
> I'm certainly not a fan of Ganglia, but this sort of unexpected breakage is very, very bad.
Re: Thinking about the next hadoop mainline release
Posted by Allen Wittenauer <aw...@apache.org>.
On Jun 17, 2011, at 7:02 PM, Rajiv Chittajallu wrote:
> Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>>
>> Actually, I was just reminded about the complete disaster that is metrics. So while it may be pseudo-stable, it isn't actually usable for anyone but Yahoo!.
>
> Did you try to use the new metrics framework? Are you complaining
> cause there is no port of ganglia metrics module to metrics V2?
Like 80-90%+ of the Hadoop sites that gather metrics, yes, we use Ganglia.
> Its pluggable and you can choose to disable it if you want to.
> All metrics and status information is available via jmx.
> Multiple sinks and filters, more importantly you can refresh
> metrics configs without restarting the process.
But first we need to switch to a new metrics infrastructure. This wouldn't have been so bad if the release notes actually had said "You know your current metrics system? Yeah, toss it because we just broke everything."
I'm certainly not a fan of Ganglia, but this sort of unexpected breakage is very, very bad.
Re: Thinking about the next hadoop mainline release
Posted by Rajiv Chittajallu <ra...@yahoo-inc.com>.
Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>
> Actually, I was just reminded about the complete disaster that is metrics. So while it may be pseudo-stable, it isn't actually usable for anyone but Yahoo!.
Did you try to use the new metrics framework? Are you complaining
cause there is no port of ganglia metrics module to metrics V2?
Its pluggable and you can choose to disable it if you want to.
All metrics and status information is available via jmx.
Multiple sinks and filters, more importantly you can refresh
metrics configs without restarting the process.
-rajive
Re: Thinking about the next hadoop mainline release
Posted by Allen Wittenauer <aw...@apache.org>.
On Jun 17, 2011, at 10:31 AM, Allen Wittenauer wrote:
>
> On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security).
>
>
> I'd consider 0.20.203 pseudo-stable.
Actually, I was just reminded about the complete disaster that is metrics. So while it may be pseudo-stable, it isn't actually usable for anyone but Yahoo!.
Re: Thinking about the next hadoop mainline release
Posted by Allen Wittenauer <aw...@apache.org>.
On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security).
I'd consider 0.20.203 pseudo-stable. It has some significant regressions on non-Linux platforms due to libhadoop.so being a dumping ground for all compiled code. On OS X, hadoop actually lies about things now. Nine months of testing is useless for those folks if the outcome is another 0.20.203.
> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
For .23, we desperately need to have libhadoop tell the Java code what it supports rather than just assuming that all the functionality is present if the library loads. I pretty much consider this a blocking issue.
Re: Thinking about the next hadoop mainline release
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
Hey Allen,
I agree with you that we should avoid future regressions like this. I think the tradeoff that got security working well on one heavily used platform was the right one for hadoop-with-security, but I'll be sure to raise such issues for discussion in the future as soon as I become aware of them.
I agree that we should think about how to generalize the security work for other platforms. That work is just waiting for someone to jump in....
On ganglia and the metrics framework, it would be great if you could put your head together with rajive and come up with a joint proposal on what docs / code is needed to fix the regression in a clean way. That sounds like something we should be thinking about fixing in the 20 line and all future releases.
thanks,
E14
On Jun 17, 2011, at 10:33 AM, Allen Wittenauer wrote:
>
> On Jun 17, 2011, at 12:36 AM, Ryan Rawson wrote:
>
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance. The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing and
>> running HDFS-918 as well. Opening 1 socket for every read just isn't
>> really scalable.
>
> Isn't "random read [on HDFS]" and "smarter sites" in the same breath an oxymoron?
>
>
Re: Thinking about the next hadoop mainline release
Posted by Allen Wittenauer <aw...@apache.org>.
On Jun 17, 2011, at 12:36 AM, Ryan Rawson wrote:
> HDFS-918 and HDFS-347 are absolutely critical for random read
> performance. The smarter sites are already running HDFS-347 (I guess
> they aren't running "Hadoop" then?), and soon they will be testing and
> running HDFS-918 as well. Opening 1 socket for every read just isn't
> really scalable.
Isn't "random read [on HDFS]" and "smarter sites" in the same breath an oxymoron?
Re: Thinking about the next hadoop mainline release
Posted by Mahadev Konar <ma...@apache.org>.
+1. I'd be happy to help in any way possible.
thanks
mahadev
On Fri, Jun 24, 2011 at 2:07 PM, Suresh Srinivas <su...@yahoo-inc.com> wrote:
> +1. Arun, I can also help you with managing the release for HDFS.
>
>
> On 6/17/11 7:15 AM, "Arun C Murthy" <ar...@yahoo-inc.com> wrote:
>
>> I volunteer to be the RM for the release since I've been leading the NG NR
>> effort.
>>
>> Are folks ok with this?
>>
>> thanks,
>> Arun
>>
>> Sent from my iPhone
>>
>> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com> wrote:
>>
>>> NG map reduce is a huge deal both in terms of making things better for
>>> users, but also in terms of unblocking the Hadoop development process.
>>>
>>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
>>>
>>>>> - Next Generation Map-Reduce [MR-279]
>>>>> - Passing most tests now and discussing merging into trunk
>>>>
>
>
--
thanks
mahadev
@mahadevkonar
Re: Thinking about the next hadoop mainline release
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Jun 17, 2011 at 7:15 AM, Arun C Murthy <ar...@yahoo-inc.com> wrote:
> I volunteer to be the RM for the release since I've been leading the NG NR effort.
>
> Are folks ok with this?
+1. It would be an honor to fix bugs for you, Arun.
-Todd
> Sent from my iPhone
>
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com> wrote:
>
>> NG map reduce is a huge deal both in terms of making things better for
>> users, but also in terms of unblocking the Hadoop development process.
>>
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
>>
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thinking about the next hadoop mainline release
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Jun 17, 2011, at 7:15 AM, Arun C Murthy wrote:
> I volunteer to be the RM for the release since I've been leading the
> NG NR effort.
>
> Are folks ok with this?
+1
sanjay
>
> thanks,
> Arun
>
> Sent from my iPhone
>
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com>
> wrote:
>
>> NG map reduce is a huge deal both in terms of making things better
>> for
>> users, but also in terms of unblocking the Hadoop development
>> process.
>>
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com>
>> wrote:
>>
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>
Re: Hadoop´s Internationalization
Posted by Ted Yu <yu...@gmail.com>.
Marcos:
Which hadoop version(s) do you plan to work on ?
>> Where I can find the sources of the docs?
In the latest TRUNK, you would find these directories:
./common/src/docs
./hdfs/src/c++/libhdfs/docs
./hdfs/src/docs
./mapreduce/src/docs
Cheers
On Sat, Jun 25, 2011 at 8:17 AM, Marcos Ortiz <ml...@uci.cu> wrote:
> OK, Owen, Where I can find the sources of the docs?
> Which is the format for the docs? DocBook, ReST, etc?
>
> El 6/25/2011 4:38 AM, Owen O'Malley escribió:
>
> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz<ml...@uci.cu> wrote:
>>
>>
>>
>>> Regards to all the list.
>>> I´m looking for a proper way to work on the internationalization of
>>> Hadoop,
>>> but I don´t know if this is a good project
>>> or if this is useful for the community, at least, I think that it would
>>> be
>>> very useful for many people that want to see the messages
>>> of the project in another language, for example, in Spanish.
>>>
>>>
>>
>> I think it would be very useful, but very time consuming project. I'd
>> suggest that you start by translating the documentation for the release
>> into
>> another language first.
>>
>> -- Owen
>>
>>
>>
>
> --
> Marcos Luís Ortíz Valmaseda
> Software Engineer (UCI)
> http://marcosluis2186.**posterous.com<http://marcosluis2186.posterous.com>
> http://twitter.com/**marcosluis2186 <http://twitter.com/marcosluis2186>
>
>
Re: Hadoop´s Internationalization
Posted by Marcos Ortiz <ml...@uci.cu>.
OK, Owen, Where I can find the sources of the docs?
Which is the format for the docs? DocBook, ReST, etc?
El 6/25/2011 4:38 AM, Owen O'Malley escribió:
> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz<ml...@uci.cu> wrote:
>
>
>> Regards to all the list.
>> I´m looking for a proper way to work on the internationalization of Hadoop,
>> but I don´t know if this is a good project
>> or if this is useful for the community, at least, I think that it would be
>> very useful for many people that want to see the messages
>> of the project in another language, for example, in Spanish.
>>
>
> I think it would be very useful, but very time consuming project. I'd
> suggest that you start by translating the documentation for the release into
> another language first.
>
> -- Owen
>
>
--
Marcos Luís Ortíz Valmaseda
Software Engineer (UCI)
http://marcosluis2186.posterous.com
http://twitter.com/marcosluis2186
Re: Hadoop´s Internationalization
Posted by Harsh J <ha...@cloudera.com>.
Having spent some time in KDE before, proper intl. could only be
achieved if string freeze dates are set and notifications are pushed
out to translator teams for a translating period of newly exported
strings (it could be an ongoing process, but before a release may be
cut there ought to be a string freeze period). Where it can't be done,
English is fallen back upon…
But KDE had large amounts of simple translation, while Hadoop has more
of paragraphs and documents that need translation (keeping API docs
and error messages aside). A different approach could be thought with
that in mind I think.
On Fri, Jul 1, 2011 at 2:56 AM, Jakob Homan <jg...@gmail.com> wrote:
> I'd be hesitant about this given our experience with the Chinese
> version of the Hadoop documents. Picking the Chinese version of
> hdfs_quota_admin_guide.xml, as an example, shows that it has not
> actually been updated (beyond svn moves and copyright years) since its
> original commit back on 12/10/08 (svn rev: 736174). I'm actually
> going to open a JIRA to remove it since it's now woefully out of data
> (and even worse than no documentation is wrong documentation). We'd
> need to make sure that there would be contributors willing and able to
> keep these documents up-to-date, which hasn't happened with the
> Chinese, even though we have quite a few Chinese-speaking
> contributors.
> -jakob
>
> On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
> <er...@yahoo-inc.com> wrote:
>> Do other apache projects have a good localization framework for error messages?
>>
>> I'd think there would be interest in starting this, although since we'd need to add the framework, this would be an investment.
>>
>> On Jun 25, 2011, at 1:38 AM, Owen O'Malley wrote:
>>
>>> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <ml...@uci.cu> wrote:
>>>
>>>> Regards to all the list.
>>>> I´m looking for a proper way to work on the internationalization of Hadoop,
>>>> but I don´t know if this is a good project
>>>> or if this is useful for the community, at least, I think that it would be
>>>> very useful for many people that want to see the messages
>>>> of the project in another language, for example, in Spanish.
>>>
>>>
>>> I think it would be very useful, but very time consuming project. I'd
>>> suggest that you start by translating the documentation for the release into
>>> another language first.
>>>
>>> -- Owen
>>
>>
>
--
Harsh J
Re: Hadoop´s Internationalization
Posted by Jakob Homan <jg...@gmail.com>.
I'd be hesitant about this given our experience with the Chinese
version of the Hadoop documents. Picking the Chinese version of
hdfs_quota_admin_guide.xml, as an example, shows that it has not
actually been updated (beyond svn moves and copyright years) since its
original commit back on 12/10/08 (svn rev: 736174). I'm actually
going to open a JIRA to remove it since it's now woefully out of data
(and even worse than no documentation is wrong documentation). We'd
need to make sure that there would be contributors willing and able to
keep these documents up-to-date, which hasn't happened with the
Chinese, even though we have quite a few Chinese-speaking
contributors.
-jakob
On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
<er...@yahoo-inc.com> wrote:
> Do other apache projects have a good localization framework for error messages?
>
> I'd think there would be interest in starting this, although since we'd need to add the framework, this would be an investment.
>
> On Jun 25, 2011, at 1:38 AM, Owen O'Malley wrote:
>
>> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <ml...@uci.cu> wrote:
>>
>>> Regards to all the list.
>>> I´m looking for a proper way to work on the internationalization of Hadoop,
>>> but I don´t know if this is a good project
>>> or if this is useful for the community, at least, I think that it would be
>>> very useful for many people that want to see the messages
>>> of the project in another language, for example, in Spanish.
>>
>>
>> I think it would be very useful, but very time consuming project. I'd
>> suggest that you start by translating the documentation for the release into
>> another language first.
>>
>> -- Owen
>
>
Re: Hadoop´s Internationalization
Posted by Luke Lu <ll...@vicaya.com>.
On Fri, Jul 1, 2011 at 8:38 AM, Steve Loughran <st...@apache.org> wrote:
> On 01/07/2011 06:44, Owen O'Malley wrote:
>>
>> On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
>> <er...@yahoo-inc.com>wrote:
>>
>>> Do other apache projects have a good localization framework for error
>>> messages?
>>>
>>
>> Java has very good localization capabilities. However, it is a huge
>> pervasive change if we want to get each and every user-facing string
>> localizable.
The pervasive change (using ResourceBundles or whatever to do the
mapping from a canonical string to whatever) is done only once.
i18n/l10n can then be improved incrementally by updating properties
(or whatever) files.
__Luke
Re: Hadoop´s Internationalization
Posted by Steve Loughran <st...@apache.org>.
On 14/07/11 22:04, Allen Wittenauer wrote:
>
> On Jul 2, 2011, at 7:23 AM, Steve Loughran wrote:
>
>> On 01/07/2011 17:23, Eric Baldeschwieler wrote:
>>> Unique error codes sounds like a very good place to start!
>>
>>
>> we could call them "URLs" and have things at the end of them
>> http://wiki.apache.org/hadoop/ConnectionRefused
>
>
> This is how Sun (I refuse to acknowledge that "other company") built SMF. It would give you a short error, but provide a URL that would point to sun.com for more info. (i.e. http://sun.com/msg/SMF-8000-05 , now locked behind Larry's boat money generator).
yeah, changing URLS is one of the weaknesses of that design. Oracle
managed to break all the javadoc links to JDK docs
>
> In the Hadoop case, we would likely want to provide a base set that folks could install and modify locally. For example, what may be an XYZ error generically may be more indicative of a problem with our local SerDe's. Putting it on the wiki would be a good thing, yes, but I worry about local tribal knowledge spilling into it.
It's strength is it stays up to date.
I've been proposing adding this to connection operations
https://issues.apache.org/jira/browse/HADOOP-7469
Re: Hadoop´s Internationalization
Posted by Allen Wittenauer <aw...@apache.org>.
On Jul 2, 2011, at 7:23 AM, Steve Loughran wrote:
> On 01/07/2011 17:23, Eric Baldeschwieler wrote:
>> Unique error codes sounds like a very good place to start!
>
>
> we could call them "URLs" and have things at the end of them
> http://wiki.apache.org/hadoop/ConnectionRefused
This is how Sun (I refuse to acknowledge that "other company") built SMF. It would give you a short error, but provide a URL that would point to sun.com for more info. (i.e. http://sun.com/msg/SMF-8000-05 , now locked behind Larry's boat money generator).
In the Hadoop case, we would likely want to provide a base set that folks could install and modify locally. For example, what may be an XYZ error generically may be more indicative of a problem with our local SerDe's. Putting it on the wiki would be a good thing, yes, but I worry about local tribal knowledge spilling into it.
I dunno. Just thinking out loud.
Re: Hadoop´s Internationalization
Posted by Steve Loughran <st...@apache.org>.
On 01/07/2011 17:23, Eric Baldeschwieler wrote:
> Unique error codes sounds like a very good place to start!
we could call them "URLs" and have things at the end of them
http://wiki.apache.org/hadoop/ConnectionRefused
Re: Hadoop´s Internationalization
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
Unique error codes sounds like a very good place to start!
Freezing the code while a hypothetical group of volunteers localizes sounds premature to me.
Are there good examples of projects with lazy localization that work well? EG The project is released in english, but there are sub-trees where additional localized content can be populated based on index numbers and such?
E14
On Jul 1, 2011, at 8:38 AM, Steve Loughran wrote:
> On 01/07/2011 06:44, Owen O'Malley wrote:
>> On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
>> <er...@yahoo-inc.com>wrote:
>>
>>> Do other apache projects have a good localization framework for error
>>> messages?
>>>
>>
>> Java has very good localization capabilities. However, it is a huge
>> pervasive change if we want to get each and every user-facing string
>> localizable.
>>
>> -- Owen
>>
> Let's be precise: Internationalisation (note the spelling) is a
> maintenance mess too. It's not so much a "one off event" as something
> you have to do every time anyone adds an error message, or you gradually
> let the percentage of i18n'd messages drop over time. Given a limitation
> of Hadoop now is that when you get near the fringes of the valid
> configuration space the messages aren't that helpful, I'd focus on those.
>
> I say "I" literally here, as it tends to me that hits these problems.
>
> In a concession to the US installed base, I will spell words like
> "datacentre" and "normalised" incorrectly for EN_GB. This is not just
> politeness, it's self interest: I added a message to Ant about an
> unknown task that said "your task is spelt wrong", and we kept on
> getting bugreps saying "you have spelled spelled wrong" that I'd close
> as "workforme, you can't spell the past tense of spelled correctly",
> until I got bored and changed it to a present-tense form that was valid
> everywhere.
>
> I18n getting started docs are good, and examples, but error messages may
> be best left as is. One possibility though is to add a unique error code
> to each one that could be indexed in each document, wiki, etc.
>
> -steve
>
>
>
Re: Hadoop´s Internationalization
Posted by Steve Loughran <st...@apache.org>.
On 01/07/2011 06:44, Owen O'Malley wrote:
> On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
> <er...@yahoo-inc.com>wrote:
>
>> Do other apache projects have a good localization framework for error
>> messages?
>>
>
> Java has very good localization capabilities. However, it is a huge
> pervasive change if we want to get each and every user-facing string
> localizable.
>
> -- Owen
>
Let's be precise: Internationalisation (note the spelling) is a
maintenance mess too. It's not so much a "one off event" as something
you have to do every time anyone adds an error message, or you gradually
let the percentage of i18n'd messages drop over time. Given a limitation
of Hadoop now is that when you get near the fringes of the valid
configuration space the messages aren't that helpful, I'd focus on those.
I say "I" literally here, as it tends to me that hits these problems.
In a concession to the US installed base, I will spell words like
"datacentre" and "normalised" incorrectly for EN_GB. This is not just
politeness, it's self interest: I added a message to Ant about an
unknown task that said "your task is spelt wrong", and we kept on
getting bugreps saying "you have spelled spelled wrong" that I'd close
as "workforme, you can't spell the past tense of spelled correctly",
until I got bored and changed it to a present-tense form that was valid
everywhere.
I18n getting started docs are good, and examples, but error messages may
be best left as is. One possibility though is to add a unique error code
to each one that could be indexed in each document, wiki, etc.
-steve
Re: Hadoop´s Internationalization
Posted by Owen O'Malley <om...@apache.org>.
On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
<er...@yahoo-inc.com>wrote:
> Do other apache projects have a good localization framework for error
> messages?
>
Java has very good localization capabilities. However, it is a huge
pervasive change if we want to get each and every user-facing string
localizable.
-- Owen
Re: Hadoop´s Internationalization
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
Do other apache projects have a good localization framework for error messages?
I'd think there would be interest in starting this, although since we'd need to add the framework, this would be an investment.
On Jun 25, 2011, at 1:38 AM, Owen O'Malley wrote:
> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <ml...@uci.cu> wrote:
>
>> Regards to all the list.
>> I´m looking for a proper way to work on the internationalization of Hadoop,
>> but I don´t know if this is a good project
>> or if this is useful for the community, at least, I think that it would be
>> very useful for many people that want to see the messages
>> of the project in another language, for example, in Spanish.
>
>
> I think it would be very useful, but very time consuming project. I'd
> suggest that you start by translating the documentation for the release into
> another language first.
>
> -- Owen
Re: Hadoop´s Internationalization
Posted by Owen O'Malley <om...@apache.org>.
On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <ml...@uci.cu> wrote:
> Regards to all the list.
> I´m looking for a proper way to work on the internationalization of Hadoop,
> but I don´t know if this is a good project
> or if this is useful for the community, at least, I think that it would be
> very useful for many people that want to see the messages
> of the project in another language, for example, in Spanish.
I think it would be very useful, but very time consuming project. I'd
suggest that you start by translating the documentation for the release into
another language first.
-- Owen
Hadoop´s Internationalization
Posted by Marcos Ortiz <ml...@uci.cu>.
Regards to all the list.
I´m looking for a proper way to work on the internationalization of
Hadoop, but I don´t know if this is a good project
or if this is useful for the community, at least, I think that it would
be very useful for many people that want to see the messages
of the project in another language, for example, in Spanish.
I would like to work on this, but I don´t know how to start on this. I
think that a first step is to translate the message of the commands more
used by administrators and developers of Hadoop: (The Hadoop fs shell,
for example).
It would be very useful to have on the source code, a directory for
earch language´s messages, but I don´t know if this could carry out
many changes on the main API.
What do you think about this?
Thanls for your time
--
Marcos Luís Ortíz Valmaseda
Software Engineer (UCI)
http://marcosluis2186.posterous.com
http://twitter.com/marcosluis2186
Re: Thinking about the next hadoop mainline release
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Jun 24, 2011 at 5:28 PM, Arun C Murthy <ar...@yahoo-inc.com> wrote:
> Thanks Suresh!
>
> Todd - I'd appreciate if you could help on some of the HBase/Performance
> jiras... thanks!
>
>
Sure thing.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thinking about the next hadoop mainline release
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Thanks Suresh!
Todd - I'd appreciate if you could help on some of the HBase/Performance jiras... thanks!
Sent from my iPhone
On Jun 24, 2011, at 2:09 PM, "Suresh Srinivas" <su...@yahoo-inc.com> wrote:
> +1. Arun, I can also help you with managing the release for HDFS.
>
>
> On 6/17/11 7:15 AM, "Arun C Murthy" <ar...@yahoo-inc.com> wrote:
>
>> I volunteer to be the RM for the release since I've been leading the NG NR
>> effort.
>>
>> Are folks ok with this?
>>
>> thanks,
>> Arun
>>
>> Sent from my iPhone
>>
>> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com> wrote:
>>
>>> NG map reduce is a huge deal both in terms of making things better for
>>> users, but also in terms of unblocking the Hadoop development process.
>>>
>>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
>>>
>>>>> - Next Generation Map-Reduce [MR-279]
>>>>> - Passing most tests now and discussing merging into trunk
>>>>
>
Re: Thinking about the next hadoop mainline release
Posted by Suresh Srinivas <su...@yahoo-inc.com>.
+1. Arun, I can also help you with managing the release for HDFS.
On 6/17/11 7:15 AM, "Arun C Murthy" <ar...@yahoo-inc.com> wrote:
> I volunteer to be the RM for the release since I've been leading the NG NR
> effort.
>
> Are folks ok with this?
>
> thanks,
> Arun
>
> Sent from my iPhone
>
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com> wrote:
>
>> NG map reduce is a huge deal both in terms of making things better for
>> users, but also in terms of unblocking the Hadoop development process.
>>
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
>>
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>
Re: Thinking about the next hadoop mainline release
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
I volunteer to be the RM for the release since I've been leading the NG NR effort.
Are folks ok with this?
thanks,
Arun
Sent from my iPhone
On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <td...@maprtech.com> wrote:
> NG map reduce is a huge deal both in terms of making things better for
> users, but also in terms of unblocking the Hadoop development process.
>
> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
>>> - Next Generation Map-Reduce [MR-279]
>>> - Passing most tests now and discussing merging into trunk
>>
Re: Thinking about the next hadoop mainline release
Posted by Ted Dunning <td...@maprtech.com>.
NG map reduce is a huge deal both in terms of making things better for
users, but also in terms of unblocking the Hadoop development process.
On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ry...@gmail.com> wrote:
> > - Next Generation Map-Reduce [MR-279]
> > - Passing most tests now and discussing merging into trunk
>
Re: Thinking about the next hadoop mainline release
Posted by Jay Booth <ja...@gmail.com>.
I can look at 1323 (hdfs-918's successor) next week/weekend and clear
the test problems, thanks Todd for updating the patch to current
trunk. 1323 is only filechannel-pooling, which is much less
disruptive than refactoring everything in the DN to be event-driven.
On Fri, Jun 17, 2011 at 10:30 AM, Brian Bockelman <bb...@cse.unl.edu> wrote:
> Hi Ryan, Eric,
>
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does seem like there's a bit of validation left.
> - HDFS-347 has a long, contentious history. However, it seems that most of the strong objections have been cleared up. Is there anyone left who objects to it, now that it doesn't appear to bypass security?
>
> Finally, I see Todd has posted HDFS-2080 claiming some sizable performance improvements. Would it be possible that could finish in time for release?
>
> As a site which heavily uses random reads and high-throughput reads, I'm very excited for this release!
>
> Brian
>
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance. The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing and
>> running HDFS-918 as well. Opening 1 socket for every read just isn't
>> really scalable.
>>
>> -ryan
>>
>> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
>> <er...@yahoo-inc.com> wrote:
>>> Hi Folks,
>>>
>>> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
>>>
>>> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
>>>
>>> - HDFS Federation [HDFS-1052]
>>> - Already merged into trunk
>>>
>>> - Next Generation Map-Reduce [MR-279]
>>> - Passing most tests now and discussing merging into trunk
>>>
>>> - The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
>>> - This is mostly done, but owen and others are doing a scrub to close out the remaining issues
>>>
>>> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
>>>
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
>>> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
>>> - Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
>>> - Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
>>>
>>> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
>>>
>>> Thanks,
>>>
>>> E14
>>>
>>>
>
>
Re: Thinking about the next hadoop mainline release
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Jun 17, 2011 at 7:30 AM, Brian Bockelman <bb...@cse.unl.edu> wrote:
>
> Hi Ryan, Eric,
>
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does seem like there's a bit of validation left.
Yes, 1323 and also 1148 would be "nice to haves", but neither is ready
to go, yet. Though I really want to improve HBase performance, I also
tend to be fairly conservative on how much testing these things should
need before getting checked in (unless they can be completely
pluggable).
The good news is we did get 941 in last week, and that's a real nice
improvement.
> - HDFS-347 has a long, contentious history. However, it seems that most of the strong objections have been cleared up. Is there anyone left who objects to it, now that it doesn't appear to bypass security?
It still has a way to go to be pushed over the finish line. I don't
foresee it happening for this release.
> Finally, I see Todd has posted HDFS-2080 claiming some sizable performance improvements. Would it be possible that could finish in time for release?
HDFS-2080 has very good bang-for-the-buck in the gains-per-complexity
ratio, especially compared to 347. It could also be made completely
pluggable, since it's just a new implementation of BlockReader. So it
might be feasible to include but not enabled by default.
But, I wouldn't block the 0.23 (or any other) release on including
these things. If they're done and look low-risk at an early enough
date, I'll do my best to convince the RM to include them, but if they
haven't had enough testing, then off to the next release with em.
-Todd
>
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>
> > HDFS-918 and HDFS-347 are absolutely critical for random read
> > performance. The smarter sites are already running HDFS-347 (I guess
> > they aren't running "Hadoop" then?), and soon they will be testing and
> > running HDFS-918 as well. Opening 1 socket for every read just isn't
> > really scalable.
> >
> > -ryan
> >
> > On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
> > <er...@yahoo-inc.com> wrote:
> >> Hi Folks,
> >>
> >> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
> >>
> >> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
> >>
> >> - HDFS Federation [HDFS-1052]
> >> - Already merged into trunk
> >>
> >> - Next Generation Map-Reduce [MR-279]
> >> - Passing most tests now and discussing merging into trunk
> >>
> >> - The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
> >> - This is mostly done, but owen and others are doing a scrub to close out the remaining issues
> >>
> >> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
> >>
> >> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
> >> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
> >> - Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
> >> - Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
> >>
> >> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
> >>
> >> Thanks,
> >>
> >> E14
> >>
> >>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thinking about the next hadoop mainline release
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Ryan & Brian,
All that needs to be included in a release branch after the branch is cut is that someone needs to convince the RM to include it. It should be fairly straight-forward.
OTOH, if it's in trunk when the branch is made this discussion is moot.
So, please provide necessary feedback to the RM when the branch is made and let's focus on a high-level goal for a next release off trunk in this thread. Makes sense?
thanks,
Arun
Sent from my iPhone
On Jun 17, 2011, at 8:01 PM, "Brian Bockelman" <bb...@cse.unl.edu> wrote:
> Hi Ryan, Eric,
>
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does seem like there's a bit of validation left.
> - HDFS-347 has a long, contentious history. However, it seems that most of the strong objections have been cleared up. Is there anyone left who objects to it, now that it doesn't appear to bypass security?
>
> Finally, I see Todd has posted HDFS-2080 claiming some sizable performance improvements. Would it be possible that could finish in time for release?
>
> As a site which heavily uses random reads and high-throughput reads, I'm very excited for this release!
>
> Brian
>
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance. The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing and
>> running HDFS-918 as well. Opening 1 socket for every read just isn't
>> really scalable.
>>
>> -ryan
>>
>> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
>> <er...@yahoo-inc.com> wrote:
>>> Hi Folks,
>>>
>>> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
>>>
>>> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
>>>
>>> - HDFS Federation [HDFS-1052]
>>> - Already merged into trunk
>>>
>>> - Next Generation Map-Reduce [MR-279]
>>> - Passing most tests now and discussing merging into trunk
>>>
>>> - The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
>>> - This is mostly done, but owen and others are doing a scrub to close out the remaining issues
>>>
>>> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
>>>
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
>>> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
>>> - Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
>>> - Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
>>>
>>> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
>>>
>>> Thanks,
>>>
>>> E14
>>>
>>>
>
Re: Thinking about the next hadoop mainline release
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hi Ryan, Eric,
Just looked at those two for the first time in awhile.
- HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does seem like there's a bit of validation left.
- HDFS-347 has a long, contentious history. However, it seems that most of the strong objections have been cleared up. Is there anyone left who objects to it, now that it doesn't appear to bypass security?
Finally, I see Todd has posted HDFS-2080 claiming some sizable performance improvements. Would it be possible that could finish in time for release?
As a site which heavily uses random reads and high-throughput reads, I'm very excited for this release!
Brian
On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
> HDFS-918 and HDFS-347 are absolutely critical for random read
> performance. The smarter sites are already running HDFS-347 (I guess
> they aren't running "Hadoop" then?), and soon they will be testing and
> running HDFS-918 as well. Opening 1 socket for every read just isn't
> really scalable.
>
> -ryan
>
> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
> <er...@yahoo-inc.com> wrote:
>> Hi Folks,
>>
>> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
>>
>> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
>>
>> - HDFS Federation [HDFS-1052]
>> - Already merged into trunk
>>
>> - Next Generation Map-Reduce [MR-279]
>> - Passing most tests now and discussing merging into trunk
>>
>> - The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
>> - This is mostly done, but owen and others are doing a scrub to close out the remaining issues
>>
>> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
>>
>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
>> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
>> - Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
>> - Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
>>
>> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
>>
>> Thanks,
>>
>> E14
>>
>>
RE: Thinking about the next hadoop mainline release
Posted by Doug Meil <do...@explorysmedical.com>.
+1 on what Ryan said.
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Friday, June 17, 2011 3:36 AM
To: general@hadoop.apache.org
Subject: Re: Thinking about the next hadoop mainline release
HDFS-918 and HDFS-347 are absolutely critical for random read performance. The smarter sites are already running HDFS-347 (I guess they aren't running "Hadoop" then?), and soon they will be testing and running HDFS-918 as well. Opening 1 socket for every read just isn't really scalable.
-ryan
On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler <er...@yahoo-inc.com> wrote:
> Hi Folks,
>
> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
>
> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
>
> - HDFS Federation [HDFS-1052]
> - Already merged into trunk
>
> - Next Generation Map-Reduce [MR-279]
> - Passing most tests now and discussing merging into trunk
>
> - The merging of our previous work on Hadoop with security into
> mainline [http://yhoo.it/i9Ww8W]
> - This is mostly done, but owen and others are doing a scrub to close
> out the remaining issues
>
> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
>
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
> - Federation - Scale up to larger clusters and the ability to
> experiment with new namenode approaches
> - Next Gen MapReduce - Scaleup, performance improvements, ability to
> experiment with new processing frameworks
>
> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
>
> Thanks,
>
> E14
>
>
Re: Thinking about the next hadoop mainline release
Posted by Ryan Rawson <ry...@gmail.com>.
HDFS-918 and HDFS-347 are absolutely critical for random read
performance. The smarter sites are already running HDFS-347 (I guess
they aren't running "Hadoop" then?), and soon they will be testing and
running HDFS-918 as well. Opening 1 socket for every read just isn't
really scalable.
-ryan
On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
<er...@yahoo-inc.com> wrote:
> Hi Folks,
>
> I'd like to start a conversation on mainline planning and the next release of Apache Hadoop beyond 0.22.
>
> The Yahoo! Hadoop team has been working hard to complete several big Hadoop projects, including:
>
> - HDFS Federation [HDFS-1052]
> - Already merged into trunk
>
> - Next Generation Map-Reduce [MR-279]
> - Passing most tests now and discussing merging into trunk
>
> - The merging of our previous work on Hadoop with security into mainline [http://yhoo.it/i9Ww8W]
> - This is mostly done, but owen and others are doing a scrub to close out the remaining issues
>
> All of these projects are now reaching a place where we would like to combine them with the good work already in 0.22 and put out a new apache release, perhaps 0.23. We think the best way to accomplish that is to finish the merge in the next few weeks and then cut a release from trunk.
>
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process. The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security). We have Yahoo!s support for this substantial investment because this new release will have a great combination of new features for small and very large sites alike:
> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
> - Federation - Scale up to larger clusters and the ability to experiment with new namenode approaches
> - Next Gen MapReduce - Scaleup, performance improvements, ability to experiment with new processing frameworks
>
> I think this effort will produce a great new Apache Hadoop release for the community. I'm starting this thread to collect feedback and hopefully folks' endorsement for merging in MR-279 and putting together this new release. Feedback please?
>
> Thanks,
>
> E14
>
>