You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Nigel Daley <nd...@yahoo-inc.com> on 2009/02/25 01:01:59 UTC
Hadoop 0.20.0
Folks,
Hadoop 0.19.1 is now available with the file append feature disabled.
It's time to talk about a Hadoop 0.20.0 release.
Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
few blockers are now almost fixed (should be next week) except for
HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
file append.
*** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
same disabling of file append in Hadoop 0.20.0 that we put in place to
get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
I will call a vote for 0.20.0 when blockers are fixed.
Cheers,
Nigel
> Folks,
>
> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
> branch has issues and a 0.19.1 release is needed.
>
> Quality issues in the changes made for the file append feature have
> prevented some from deploying Hadoop 0.19. One of these changes
> (sync) has now been "fixed" by reducing its semantics in Hadoop
> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
> branch.
>
> I would like to propose that we apply this same "fix" to sync in
> 0.19.1 and 0.20.0. Since append requires the full semantics of
> sync, I propose we also disable append (perhaps throw
> UnsupportedOperationException from API?). Yes, this would
> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
> We can then take the time needed to fix append properly in 0.21.0.
>
> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>
> Nigel
RE: Hadoop 0.20.0
Posted by Jonathan Gray <jl...@streamy.com>.
-1
This is a huge deal for me as an HBase user. And as someone who does
significant consulting and evangelism for HBase, there is probably not a
more necessary feature at this stage of the project.
Writes in HBase are done in memory, and we also write each edit to the HLog
which is periodically pushed to HDFS to prevent data loss should the node go
down in a way that does not allow us to flush everything properly. Without
appends, we have to create a new file each time we "checkpoint". In
practice, this was every 10,000 edits.
With appends, we only need to sync()/flush() to the same file. In practice,
the default number of edits between flushes became 200, and for those who
require absolutely no data loss it can now be set to 1 without writing a new
hdfs file each time, meaning slow but tolerable performance.
The upcoming 0.20.0 release of HBase is going to bring with it massive
performance boosts and should put us at or above the performance of our
competitors. Data loss is killer and is going to be enough to deter new and
existing users; at that point, we're no longer a fault-tolerant system.
I typically do not vote strongly for something if I'm not able to contribute
myself, but my impression from our testing over in HBase is that things are
not far off from working satisfactorily for our use case. If it's only
partially working, I'd much prefer it be slapped with warnings about not
being fully-functional than pulling what does work out from the next
release.
Jonathan Gray
> -----Original Message-----
> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> Sent: Tuesday, February 24, 2009 4:02 PM
> To: core-dev@hadoop.apache.org
> Subject: Hadoop 0.20.0
>
> Folks,
>
> Hadoop 0.19.1 is now available with the file append feature disabled.
> It's time to talk about a Hadoop 0.20.0 release.
>
> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
> few blockers are now almost fixed (should be next week) except for
> HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
> file append.
>
> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
> same disabling of file append in Hadoop 0.20.0 that we put in place to
> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>
> I will call a vote for 0.20.0 when blockers are fixed.
>
> Cheers,
> Nigel
>
>
> > Folks,
> >
> > Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
> > branch has issues and a 0.19.1 release is needed.
> >
> > Quality issues in the changes made for the file append feature have
> > prevented some from deploying Hadoop 0.19. One of these changes
> > (sync) has now been "fixed" by reducing its semantics in Hadoop
> > 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
> > branch.
> >
> > I would like to propose that we apply this same "fix" to sync in
> > 0.19.1 and 0.20.0. Since append requires the full semantics of
> > sync, I propose we also disable append (perhaps throw
> > UnsupportedOperationException from API?). Yes, this would
> > unfortunately be an incompatible change between 0.19.0 and 0.19.1.
> > We can then take the time needed to fix append properly in 0.21.0.
> >
> > I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
> >
> > Nigel
Re: Hadoop 0.20.0
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
+1
Hi Folks,
I don't think vetoing releases until the append issue is resolved is
constructive. I'd like to invite folks who want to fix append to work
with us on that and support releasing 20 without it in parallel.
Please keep in mind the spirit of this project. Putting your shoulder
to the wheel gets features added to the project. Vetoing a release
will not get anything added to it. It will simply delay getting those
features we've all agreed work tested and released. Resolving the
append issues is going to take a big investment. We all want to see
that happen. But we also need to provide the whole hadoop community
with stable releases while we do the hard work needed to really solve
this problem.
Why are we taking this position? The current append design has proven
fix resistant. We've deployed fix after fix and have continued to see
new append bugs that have caused data loss and cluster down time. We
moved to disable append in our clusters because that was the only path
we could see to delivering a stable system to our users. We believe
the only way to solve this problem is to acknowledge that the append
code is a critical section of the code that needs to be designed,
documented and tested thoroughly if we expect it to function well.
Given the pain the append issue has caused us, we are going to -1
"quick fixes" to it. Data loss and corruption are serious issues.
We've been burned.
I hope folks will consider this and both support a 20 release without
append and also work with us to really fix append. We promise to work
hard with the community to deliver a 20.x or 21 release with append.
Thanks,
E14
---
eric14 a.k.a. Eric Baldeschwieler
vp grid computing
Yahoo! Inc.
On Feb 25, 2009, at 7:34 PM, Nigel Daley wrote:
>
> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>
>> Hi Jim,
>>
>> I can understand your problem. I can probably whip out a fix for
>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
>> nice if
>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
>> discuss and
>> review the patches/fixes.
>
> "Whipping out a patch" doesn't give me any confidence that this
> feature will be fixed properly. We're building a file system. Data
> reliability and accuracy are absolutely key. We know that this
> feature has been very lightly tested.
>
>> Nigel: wht is the proposed deadline for 0.20?
>
> March 6.
>
> Nige
>
>> thanks,
>> dhruba
>>
>>
>>
>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>> Jim.Kellerman@microsoft.com> wrote:
>>
>>> --1
>>>
>>> HBase really needs 4379. My testing to date indicates that it does
>>> work
>>> (although I have a bit more testing to do).
>>>
>>> I was ok with not putting it into 0.19.1 provided it was in 0.19.2
>>> and
>>> 0.20.0.
>>>
>>> It's a big problem for us now and is hurting our ability to keep our
>>> community alive. (They will go to Cassandra or something else to
>>> ensure
>>> reliability).
>>>
>>>
>>>> -----Original Message-----
>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>> To: core-dev@hadoop.apache.org
>>>> Subject: Hadoop 0.20.0
>>>>
>>>> Folks,
>>>>
>>>> Hadoop 0.19.1 is now available with the file append feature
>>>> disabled.
>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>
>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
>>>> last
>>>> few blockers are now almost fixed (should be next week) except for
>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>> implement
>>>> file append.
>>>>
>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
>>>> the
>>>> same disabling of file append in Hadoop 0.20.0 that we put in
>>>> place to
>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>
>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>
>>>> Cheers,
>>>> Nigel
>>>>
>>>>
>>>>> Folks,
>>>>>
>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
>>>>> 0.19
>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>
>>>>> Quality issues in the changes made for the file append feature
>>>>> have
>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>> branch.
>>>>>
>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>> sync, I propose we also disable append (perhaps throw
>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>>
>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>>
>>>>> Nigel
>>>
>>>
>
Re: Hadoop 0.20.0
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
+1 for config option for both 0.19 and 0.20.
Raghu.
Dhruba Borthakur wrote:
> I posted a patch for HADOOP-5332. I am suggesting that this patch be applied
> into the 0.19, 0.20 and trunk. This patch switches off "append" by default,
> but it can be switched on by setting the config parameter
> dfs.support.append. This does not mean that "append" is bug free in the
> code, it just allows developers to continue testing with append
> functionality till the bugs are fixed.
>
> thanks,
> dhruba
>
> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yh...@yahoo-inc.com>wrote:
>
>> +1 for HADOOP-5332. I am in the same position as Brian, as an outside
>> observer. This will help us to move on Hadoop 0.20 which has a lot of other
>> features as well that users are asking for.
>>
>> Thanks
>> hemanth
Re: Hadoop 0.20.0
Posted by Dhruba Borthakur <dh...@gmail.com>.
I would like to make HADOOP-5332 be part of the 0.19, 0.20 and trunk. This
ensures that "append" is switched off by default. At the same time, we would
need patches for HADOOP-4739, HADOOP-4663 and HADOOP-5027. These three are
critical to support "appends".
A series of offline discussions have been summarised in HADOOP-4663. I have
not yet got comments on this summary but I am already working on it and will
post a patch early next week.
thanks,
dhruba
On Fri, Feb 27, 2009 at 9:20 AM, Doug Judd <do...@zvents.com> wrote:
> I'd like to second that. I think it would be good to have the database
> elevated to a first class use case for HDFS. Getting fsync() working
> properly is critical for HBase, Hypertable, or any database built on top of
> HDFS.
>
> - Doug
>
> On Fri, Feb 27, 2009 at 9:08 AM, Jim Kellerman (POWERSET) <
> Jim.Kellerman@microsoft.com> wrote:
>
> > I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
> > We are really hurting without it.
> >
> > ---
> > Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> >
> > > -----Original Message-----
> > > From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> > > Sent: Thursday, February 26, 2009 9:13 PM
> > > To: core-dev@hadoop.apache.org
> > > Subject: Re: Hadoop 0.20.0
> > >
> > > Thanks Jim.
> > >
> > > Dhruba, can we move
> > > https://issues.apache.org/jira/browse/HADOOP-4379
> > > to 0.21.0?
> > >
> > > Nige
> > >
> > > On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:
> > >
> > > > With the availability of HADOOP-5332 I remove my objection.
> > > >
> > > >> -----Original Message-----
> > > >> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> > > >> Sent: Wednesday, February 25, 2009 9:32 PM
> > > >> To: core-dev@hadoop.apache.org
> > > >> Subject: Re: Hadoop 0.20.0
> > > >>
> > > >> I posted a patch for HADOOP-5332. I am suggesting that this patch be
> > > >> applied
> > > >> into the 0.19, 0.20 and trunk. This patch switches off "append" by
> > > >> default,
> > > >> but it can be switched on by setting the config parameter
> > > >> dfs.support.append. This does not mean that "append" is bug free in
> > > >> the
> > > >> code, it just allows developers to continue testing with append
> > > >> functionality till the bugs are fixed.
> > > >>
> > > >> thanks,
> > > >> dhruba
> > > >>
> > > >> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
> > > >> inc.com>wrote:
> > > >>
> > > >>> +1 for HADOOP-5332. I am in the same position as Brian, as an
> > > >>> outside
> > > >>> observer. This will help us to move on Hadoop 0.20 which has a lot
> > > >>> of
> > > >> other
> > > >>> features as well that users are asking for.
> > > >>>
> > > >>> Thanks
> > > >>> hemanth
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
> > > >>>>
> > > >>>>
> > > >>>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
> > > >>>>>
> > > >>>>> "Whipping out a patch" says nothing about its reliability.
> > > >>>>>>
> > > >>>>>> i would like some focus from the developer's community to
> > > >>>>>> properly
> > > >> fix
> > > >>>>>> this
> > > >>>>>> issue. I am willing to spend as much as time it takes ot get it
> > > >> fixed
> > > >>>>>> the
> > > >>>>>> right way, I but I would like even more constructive engagement
> > > >> from
> > > >>>>>> more
> > > >>>>>> people to get this one right. May I request you to see if you
> can
> > > >>>>>> volunteer
> > > >>>>>> to spend some time testing some of this code at scale ?(I have
> > > >> access to
> > > >>>>>> 10
> > > >>>>>> machines only for testing).
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>> Dhruba, can you define "testing some of this code at scale"? Do
> you
> > > >> simply
> > > >>>> need access or folks who can run challenging jobs? Scaring up
> > > >>>> access
> > > >> to the
> > > >>>> cluster can be easy, but admin / user time isn't really available.
> > > >>>>
> > > >>>> Sorry, I can't commit any time/resources to this right now.
> Perhaps
> > > >> some
> > > >>>>> hbase folks can. In the meantime, can we make append
> > > >>>>> configurable in
> > > >> 0.19.2
> > > >>>>> and 0.20.0? I filed
> > > >>>>> https://issues.apache.org/jira/browse/HADOOP-5332
> > > >>>>>
> > > >>>>
> > > >>>> As an outside, irrelevant observer, I think this is a really good
> > > >>>> compromise. Helps out HBase but also would help prevent rushing.
> > > >>>>
> > > >>>> Brian
> > > >>>>
> > > >>>>
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> Nige
> > > >>>>>
> > > >>>>>
> > > >>>>>>
> > > >>>>>> thanks
> > > >>>>>> dhruba
> > > >>>>>>
> > > >>>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <
> > ndaley@yahoo-inc.com
> > > >>>>>> >
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Jim,
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> I can understand your problem. I can probably whip out a fix
> > > >>>>>>>> for
> > > >>>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would
> > > >>>>>>>> be
> > > >> nice
> > > >>>>>>>> if
> > > >>>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
> > > >> discuss
> > > >>>>>>>> and
> > > >>>>>>>> review the patches/fixes.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>> "Whipping out a patch" doesn't give me any confidence that this
> > > >> feature
> > > >>>>>>> will be fixed properly. We're building a file system. Data
> > > >> reliability
> > > >>>>>>> and
> > > >>>>>>> accuracy are absolutely key. We know that this feature has been
> > > >> very
> > > >>>>>>> lightly tested.
> > > >>>>>>>
> > > >>>>>>> Nigel: wht is the proposed deadline for 0.20?
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>> March 6.
> > > >>>>>>>
> > > >>>>>>> Nige
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> thanks,
> > > >>>>>>>
> > > >>>>>>>> dhruba
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
> > > >>>>>>>> Jim.Kellerman@microsoft.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>> --1
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> HBase really needs 4379. My testing to date indicates that it
> > > >> does
> > > >>>>>>>>> work
> > > >>>>>>>>> (although I have a bit more testing to do).
> > > >>>>>>>>>
> > > >>>>>>>>> I was ok with not putting it into 0.19.1 provided it was in
> > > >> 0.19.2
> > > >>>>>>>>> and
> > > >>>>>>>>> 0.20.0.
> > > >>>>>>>>>
> > > >>>>>>>>> It's a big problem for us now and is hurting our ability to
> > > >>>>>>>>> keep
> > > >> our
> > > >>>>>>>>> community alive. (They will go to Cassandra or something
> > > >>>>>>>>> else to
> > > >>>>>>>>> ensure
> > > >>>>>>>>> reliability).
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> -----Original Message-----
> > > >>>>>>>>>
> > > >>>>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> > > >>>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
> > > >>>>>>>>>> To: core-dev@hadoop.apache.org
> > > >>>>>>>>>> Subject: Hadoop 0.20.0
> > > >>>>>>>>>>
> > > >>>>>>>>>> Folks,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hadoop 0.19.1 is now available with the file append feature
> > > >>>>>>>>>> disabled.
> > > >>>>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago.
> > > >>>>>>>>>> The
> > > >> last
> > > >>>>>>>>>> few blockers are now almost fixed (should be next week)
> > > >>>>>>>>>> except
> > > >> for
> > > >>>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
> > > >>>>>>>>>> implement
> > > >>>>>>>>>> file append.
> > > >>>>>>>>>>
> > > >>>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
> > > >> apply
> > > >>>>>>>>>> the
> > > >>>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put
> in
> > > >> place
> > > >>>>>>>>>> to
> > > >>>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
> > > >>>>>>>>>>
> > > >>>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Nigel
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Folks,
> > > >>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly,
> > > >>>>>>>>>>> the
> > > >> 0.19
> > > >>>>>>>>>>> branch has issues and a 0.19.1 release is needed.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Quality issues in the changes made for the file append
> > > >>>>>>>>>>> feature
> > > >> have
> > > >>>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these
> > > >> changes
> > > >>>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in
> > > >> Hadoop
> > > >>>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the
> > > >>>>>>>>>>> 0.18
> > > >>>>>>>>>>> branch.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I would like to propose that we apply this same "fix" to
> > > >>>>>>>>>>> sync
> > > >> in
> > > >>>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full
> > > >>>>>>>>>>> semantics of
> > > >>>>>>>>>>> sync, I propose we also disable append (perhaps throw
> > > >>>>>>>>>>> UnsupportedOperationException from API?). Yes, this would
> > > >>>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and
> > > >> 0.19.1.
> > > >>>>>>>>>>> We can then take the time needed to fix append properly in
> > > >> 0.21.0.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
> > > >> fixed.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Nigel
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>>
> > >
> >
> >
>
Re: Hadoop 0.20.0
Posted by Doug Judd <do...@zvents.com>.
I'd like to second that. I think it would be good to have the database
elevated to a first class use case for HDFS. Getting fsync() working
properly is critical for HBase, Hypertable, or any database built on top of
HDFS.
- Doug
On Fri, Feb 27, 2009 at 9:08 AM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:
> I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
> We are really hurting without it.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
> > -----Original Message-----
> > From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> > Sent: Thursday, February 26, 2009 9:13 PM
> > To: core-dev@hadoop.apache.org
> > Subject: Re: Hadoop 0.20.0
> >
> > Thanks Jim.
> >
> > Dhruba, can we move
> > https://issues.apache.org/jira/browse/HADOOP-4379
> > to 0.21.0?
> >
> > Nige
> >
> > On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:
> >
> > > With the availability of HADOOP-5332 I remove my objection.
> > >
> > >> -----Original Message-----
> > >> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> > >> Sent: Wednesday, February 25, 2009 9:32 PM
> > >> To: core-dev@hadoop.apache.org
> > >> Subject: Re: Hadoop 0.20.0
> > >>
> > >> I posted a patch for HADOOP-5332. I am suggesting that this patch be
> > >> applied
> > >> into the 0.19, 0.20 and trunk. This patch switches off "append" by
> > >> default,
> > >> but it can be switched on by setting the config parameter
> > >> dfs.support.append. This does not mean that "append" is bug free in
> > >> the
> > >> code, it just allows developers to continue testing with append
> > >> functionality till the bugs are fixed.
> > >>
> > >> thanks,
> > >> dhruba
> > >>
> > >> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
> > >> inc.com>wrote:
> > >>
> > >>> +1 for HADOOP-5332. I am in the same position as Brian, as an
> > >>> outside
> > >>> observer. This will help us to move on Hadoop 0.20 which has a lot
> > >>> of
> > >> other
> > >>> features as well that users are asking for.
> > >>>
> > >>> Thanks
> > >>> hemanth
> > >>>
> > >>>
> > >>>
> > >>>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
> > >>>>
> > >>>>
> > >>>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
> > >>>>>
> > >>>>> "Whipping out a patch" says nothing about its reliability.
> > >>>>>>
> > >>>>>> i would like some focus from the developer's community to
> > >>>>>> properly
> > >> fix
> > >>>>>> this
> > >>>>>> issue. I am willing to spend as much as time it takes ot get it
> > >> fixed
> > >>>>>> the
> > >>>>>> right way, I but I would like even more constructive engagement
> > >> from
> > >>>>>> more
> > >>>>>> people to get this one right. May I request you to see if you can
> > >>>>>> volunteer
> > >>>>>> to spend some time testing some of this code at scale ?(I have
> > >> access to
> > >>>>>> 10
> > >>>>>> machines only for testing).
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>> Dhruba, can you define "testing some of this code at scale"? Do you
> > >> simply
> > >>>> need access or folks who can run challenging jobs? Scaring up
> > >>>> access
> > >> to the
> > >>>> cluster can be easy, but admin / user time isn't really available.
> > >>>>
> > >>>> Sorry, I can't commit any time/resources to this right now. Perhaps
> > >> some
> > >>>>> hbase folks can. In the meantime, can we make append
> > >>>>> configurable in
> > >> 0.19.2
> > >>>>> and 0.20.0? I filed
> > >>>>> https://issues.apache.org/jira/browse/HADOOP-5332
> > >>>>>
> > >>>>
> > >>>> As an outside, irrelevant observer, I think this is a really good
> > >>>> compromise. Helps out HBase but also would help prevent rushing.
> > >>>>
> > >>>> Brian
> > >>>>
> > >>>>
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Nige
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>> thanks
> > >>>>>> dhruba
> > >>>>>>
> > >>>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <
> ndaley@yahoo-inc.com
> > >>>>>> >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
> > >>>>>>>
> > >>>>>>> Hi Jim,
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> I can understand your problem. I can probably whip out a fix
> > >>>>>>>> for
> > >>>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would
> > >>>>>>>> be
> > >> nice
> > >>>>>>>> if
> > >>>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
> > >> discuss
> > >>>>>>>> and
> > >>>>>>>> review the patches/fixes.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> "Whipping out a patch" doesn't give me any confidence that this
> > >> feature
> > >>>>>>> will be fixed properly. We're building a file system. Data
> > >> reliability
> > >>>>>>> and
> > >>>>>>> accuracy are absolutely key. We know that this feature has been
> > >> very
> > >>>>>>> lightly tested.
> > >>>>>>>
> > >>>>>>> Nigel: wht is the proposed deadline for 0.20?
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> March 6.
> > >>>>>>>
> > >>>>>>> Nige
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> thanks,
> > >>>>>>>
> > >>>>>>>> dhruba
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
> > >>>>>>>> Jim.Kellerman@microsoft.com> wrote:
> > >>>>>>>>
> > >>>>>>>> --1
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> HBase really needs 4379. My testing to date indicates that it
> > >> does
> > >>>>>>>>> work
> > >>>>>>>>> (although I have a bit more testing to do).
> > >>>>>>>>>
> > >>>>>>>>> I was ok with not putting it into 0.19.1 provided it was in
> > >> 0.19.2
> > >>>>>>>>> and
> > >>>>>>>>> 0.20.0.
> > >>>>>>>>>
> > >>>>>>>>> It's a big problem for us now and is hurting our ability to
> > >>>>>>>>> keep
> > >> our
> > >>>>>>>>> community alive. (They will go to Cassandra or something
> > >>>>>>>>> else to
> > >>>>>>>>> ensure
> > >>>>>>>>> reliability).
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>>
> > >>>>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> > >>>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
> > >>>>>>>>>> To: core-dev@hadoop.apache.org
> > >>>>>>>>>> Subject: Hadoop 0.20.0
> > >>>>>>>>>>
> > >>>>>>>>>> Folks,
> > >>>>>>>>>>
> > >>>>>>>>>> Hadoop 0.19.1 is now available with the file append feature
> > >>>>>>>>>> disabled.
> > >>>>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
> > >>>>>>>>>>
> > >>>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago.
> > >>>>>>>>>> The
> > >> last
> > >>>>>>>>>> few blockers are now almost fixed (should be next week)
> > >>>>>>>>>> except
> > >> for
> > >>>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
> > >>>>>>>>>> implement
> > >>>>>>>>>> file append.
> > >>>>>>>>>>
> > >>>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
> > >> apply
> > >>>>>>>>>> the
> > >>>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
> > >> place
> > >>>>>>>>>> to
> > >>>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
> > >>>>>>>>>>
> > >>>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Nigel
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Folks,
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly,
> > >>>>>>>>>>> the
> > >> 0.19
> > >>>>>>>>>>> branch has issues and a 0.19.1 release is needed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Quality issues in the changes made for the file append
> > >>>>>>>>>>> feature
> > >> have
> > >>>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these
> > >> changes
> > >>>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in
> > >> Hadoop
> > >>>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the
> > >>>>>>>>>>> 0.18
> > >>>>>>>>>>> branch.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I would like to propose that we apply this same "fix" to
> > >>>>>>>>>>> sync
> > >> in
> > >>>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full
> > >>>>>>>>>>> semantics of
> > >>>>>>>>>>> sync, I propose we also disable append (perhaps throw
> > >>>>>>>>>>> UnsupportedOperationException from API?). Yes, this would
> > >>>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and
> > >> 0.19.1.
> > >>>>>>>>>>> We can then take the time needed to fix append properly in
> > >> 0.21.0.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
> > >> fixed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Nigel
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> >
>
>
RE: Hadoop 0.20.0
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
We are really hurting without it.
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> Sent: Thursday, February 26, 2009 9:13 PM
> To: core-dev@hadoop.apache.org
> Subject: Re: Hadoop 0.20.0
>
> Thanks Jim.
>
> Dhruba, can we move
> https://issues.apache.org/jira/browse/HADOOP-4379
> to 0.21.0?
>
> Nige
>
> On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:
>
> > With the availability of HADOOP-5332 I remove my objection.
> >
> >> -----Original Message-----
> >> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> >> Sent: Wednesday, February 25, 2009 9:32 PM
> >> To: core-dev@hadoop.apache.org
> >> Subject: Re: Hadoop 0.20.0
> >>
> >> I posted a patch for HADOOP-5332. I am suggesting that this patch be
> >> applied
> >> into the 0.19, 0.20 and trunk. This patch switches off "append" by
> >> default,
> >> but it can be switched on by setting the config parameter
> >> dfs.support.append. This does not mean that "append" is bug free in
> >> the
> >> code, it just allows developers to continue testing with append
> >> functionality till the bugs are fixed.
> >>
> >> thanks,
> >> dhruba
> >>
> >> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
> >> inc.com>wrote:
> >>
> >>> +1 for HADOOP-5332. I am in the same position as Brian, as an
> >>> outside
> >>> observer. This will help us to move on Hadoop 0.20 which has a lot
> >>> of
> >> other
> >>> features as well that users are asking for.
> >>>
> >>> Thanks
> >>> hemanth
> >>>
> >>>
> >>>
> >>>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
> >>>>
> >>>>
> >>>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
> >>>>>
> >>>>> "Whipping out a patch" says nothing about its reliability.
> >>>>>>
> >>>>>> i would like some focus from the developer's community to
> >>>>>> properly
> >> fix
> >>>>>> this
> >>>>>> issue. I am willing to spend as much as time it takes ot get it
> >> fixed
> >>>>>> the
> >>>>>> right way, I but I would like even more constructive engagement
> >> from
> >>>>>> more
> >>>>>> people to get this one right. May I request you to see if you can
> >>>>>> volunteer
> >>>>>> to spend some time testing some of this code at scale ?(I have
> >> access to
> >>>>>> 10
> >>>>>> machines only for testing).
> >>>>>>
> >>>>>
> >>>>>
> >>>> Dhruba, can you define "testing some of this code at scale"? Do you
> >> simply
> >>>> need access or folks who can run challenging jobs? Scaring up
> >>>> access
> >> to the
> >>>> cluster can be easy, but admin / user time isn't really available.
> >>>>
> >>>> Sorry, I can't commit any time/resources to this right now. Perhaps
> >> some
> >>>>> hbase folks can. In the meantime, can we make append
> >>>>> configurable in
> >> 0.19.2
> >>>>> and 0.20.0? I filed
> >>>>> https://issues.apache.org/jira/browse/HADOOP-5332
> >>>>>
> >>>>
> >>>> As an outside, irrelevant observer, I think this is a really good
> >>>> compromise. Helps out HBase but also would help prevent rushing.
> >>>>
> >>>> Brian
> >>>>
> >>>>
> >>>>>
> >>>>> Cheers,
> >>>>> Nige
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> thanks
> >>>>>> dhruba
> >>>>>>
> >>>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com
> >>>>>> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
> >>>>>>>
> >>>>>>> Hi Jim,
> >>>>>>>
> >>>>>>>>
> >>>>>>>> I can understand your problem. I can probably whip out a fix
> >>>>>>>> for
> >>>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would
> >>>>>>>> be
> >> nice
> >>>>>>>> if
> >>>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
> >> discuss
> >>>>>>>> and
> >>>>>>>> review the patches/fixes.
> >>>>>>>>
> >>>>>>>>
> >>>>>>> "Whipping out a patch" doesn't give me any confidence that this
> >> feature
> >>>>>>> will be fixed properly. We're building a file system. Data
> >> reliability
> >>>>>>> and
> >>>>>>> accuracy are absolutely key. We know that this feature has been
> >> very
> >>>>>>> lightly tested.
> >>>>>>>
> >>>>>>> Nigel: wht is the proposed deadline for 0.20?
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> March 6.
> >>>>>>>
> >>>>>>> Nige
> >>>>>>>
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>>
> >>>>>>>> dhruba
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
> >>>>>>>> Jim.Kellerman@microsoft.com> wrote:
> >>>>>>>>
> >>>>>>>> --1
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> HBase really needs 4379. My testing to date indicates that it
> >> does
> >>>>>>>>> work
> >>>>>>>>> (although I have a bit more testing to do).
> >>>>>>>>>
> >>>>>>>>> I was ok with not putting it into 0.19.1 provided it was in
> >> 0.19.2
> >>>>>>>>> and
> >>>>>>>>> 0.20.0.
> >>>>>>>>>
> >>>>>>>>> It's a big problem for us now and is hurting our ability to
> >>>>>>>>> keep
> >> our
> >>>>>>>>> community alive. (They will go to Cassandra or something
> >>>>>>>>> else to
> >>>>>>>>> ensure
> >>>>>>>>> reliability).
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>>
> >>>>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> >>>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
> >>>>>>>>>> To: core-dev@hadoop.apache.org
> >>>>>>>>>> Subject: Hadoop 0.20.0
> >>>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> Hadoop 0.19.1 is now available with the file append feature
> >>>>>>>>>> disabled.
> >>>>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
> >>>>>>>>>>
> >>>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago.
> >>>>>>>>>> The
> >> last
> >>>>>>>>>> few blockers are now almost fixed (should be next week)
> >>>>>>>>>> except
> >> for
> >>>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
> >>>>>>>>>> implement
> >>>>>>>>>> file append.
> >>>>>>>>>>
> >>>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
> >> apply
> >>>>>>>>>> the
> >>>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
> >> place
> >>>>>>>>>> to
> >>>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
> >>>>>>>>>>
> >>>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Nigel
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly,
> >>>>>>>>>>> the
> >> 0.19
> >>>>>>>>>>> branch has issues and a 0.19.1 release is needed.
> >>>>>>>>>>>
> >>>>>>>>>>> Quality issues in the changes made for the file append
> >>>>>>>>>>> feature
> >> have
> >>>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these
> >> changes
> >>>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in
> >> Hadoop
> >>>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the
> >>>>>>>>>>> 0.18
> >>>>>>>>>>> branch.
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to propose that we apply this same "fix" to
> >>>>>>>>>>> sync
> >> in
> >>>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full
> >>>>>>>>>>> semantics of
> >>>>>>>>>>> sync, I propose we also disable append (perhaps throw
> >>>>>>>>>>> UnsupportedOperationException from API?). Yes, this would
> >>>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and
> >> 0.19.1.
> >>>>>>>>>>> We can then take the time needed to fix append properly in
> >> 0.21.0.
> >>>>>>>>>>>
> >>>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
> >> fixed.
> >>>>>>>>>>>
> >>>>>>>>>>> Nigel
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
>
Re: Hadoop 0.20.0
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Thanks Jim.
Dhruba, can we move
https://issues.apache.org/jira/browse/HADOOP-4379
to 0.21.0?
Nige
On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:
> With the availability of HADOOP-5332 I remove my objection.
>
>> -----Original Message-----
>> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
>> Sent: Wednesday, February 25, 2009 9:32 PM
>> To: core-dev@hadoop.apache.org
>> Subject: Re: Hadoop 0.20.0
>>
>> I posted a patch for HADOOP-5332. I am suggesting that this patch be
>> applied
>> into the 0.19, 0.20 and trunk. This patch switches off "append" by
>> default,
>> but it can be switched on by setting the config parameter
>> dfs.support.append. This does not mean that "append" is bug free in
>> the
>> code, it just allows developers to continue testing with append
>> functionality till the bugs are fixed.
>>
>> thanks,
>> dhruba
>>
>> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
>> inc.com>wrote:
>>
>>> +1 for HADOOP-5332. I am in the same position as Brian, as an
>>> outside
>>> observer. This will help us to move on Hadoop 0.20 which has a lot
>>> of
>> other
>>> features as well that users are asking for.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>>
>>>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
>>>>
>>>>
>>>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
>>>>>
>>>>> "Whipping out a patch" says nothing about its reliability.
>>>>>>
>>>>>> i would like some focus from the developer's community to
>>>>>> properly
>> fix
>>>>>> this
>>>>>> issue. I am willing to spend as much as time it takes ot get it
>> fixed
>>>>>> the
>>>>>> right way, I but I would like even more constructive engagement
>> from
>>>>>> more
>>>>>> people to get this one right. May I request you to see if you can
>>>>>> volunteer
>>>>>> to spend some time testing some of this code at scale ?(I have
>> access to
>>>>>> 10
>>>>>> machines only for testing).
>>>>>>
>>>>>
>>>>>
>>>> Dhruba, can you define "testing some of this code at scale"? Do you
>> simply
>>>> need access or folks who can run challenging jobs? Scaring up
>>>> access
>> to the
>>>> cluster can be easy, but admin / user time isn't really available.
>>>>
>>>> Sorry, I can't commit any time/resources to this right now. Perhaps
>> some
>>>>> hbase folks can. In the meantime, can we make append
>>>>> configurable in
>> 0.19.2
>>>>> and 0.20.0? I filed
>>>>> https://issues.apache.org/jira/browse/HADOOP-5332
>>>>>
>>>>
>>>> As an outside, irrelevant observer, I think this is a really good
>>>> compromise. Helps out HBase but also would help prevent rushing.
>>>>
>>>> Brian
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> Nige
>>>>>
>>>>>
>>>>>>
>>>>>> thanks
>>>>>> dhruba
>>>>>>
>>>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com
>>>>>> >
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>>>>>>>
>>>>>>> Hi Jim,
>>>>>>>
>>>>>>>>
>>>>>>>> I can understand your problem. I can probably whip out a fix
>>>>>>>> for
>>>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would
>>>>>>>> be
>> nice
>>>>>>>> if
>>>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
>> discuss
>>>>>>>> and
>>>>>>>> review the patches/fixes.
>>>>>>>>
>>>>>>>>
>>>>>>> "Whipping out a patch" doesn't give me any confidence that this
>> feature
>>>>>>> will be fixed properly. We're building a file system. Data
>> reliability
>>>>>>> and
>>>>>>> accuracy are absolutely key. We know that this feature has been
>> very
>>>>>>> lightly tested.
>>>>>>>
>>>>>>> Nigel: wht is the proposed deadline for 0.20?
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> March 6.
>>>>>>>
>>>>>>> Nige
>>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>>> dhruba
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>>>>>>>> Jim.Kellerman@microsoft.com> wrote:
>>>>>>>>
>>>>>>>> --1
>>>>>>>>
>>>>>>>>>
>>>>>>>>> HBase really needs 4379. My testing to date indicates that it
>> does
>>>>>>>>> work
>>>>>>>>> (although I have a bit more testing to do).
>>>>>>>>>
>>>>>>>>> I was ok with not putting it into 0.19.1 provided it was in
>> 0.19.2
>>>>>>>>> and
>>>>>>>>> 0.20.0.
>>>>>>>>>
>>>>>>>>> It's a big problem for us now and is hurting our ability to
>>>>>>>>> keep
>> our
>>>>>>>>> community alive. (They will go to Cassandra or something
>>>>>>>>> else to
>>>>>>>>> ensure
>>>>>>>>> reliability).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>>
>>>>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>>>>>>>> To: core-dev@hadoop.apache.org
>>>>>>>>>> Subject: Hadoop 0.20.0
>>>>>>>>>>
>>>>>>>>>> Folks,
>>>>>>>>>>
>>>>>>>>>> Hadoop 0.19.1 is now available with the file append feature
>>>>>>>>>> disabled.
>>>>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>>>>>>>
>>>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago.
>>>>>>>>>> The
>> last
>>>>>>>>>> few blockers are now almost fixed (should be next week)
>>>>>>>>>> except
>> for
>>>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>>>>>>>> implement
>>>>>>>>>> file append.
>>>>>>>>>>
>>>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
>> apply
>>>>>>>>>> the
>>>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
>> place
>>>>>>>>>> to
>>>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>>>>>>>
>>>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Nigel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Folks,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly,
>>>>>>>>>>> the
>> 0.19
>>>>>>>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>>>>>>>
>>>>>>>>>>> Quality issues in the changes made for the file append
>>>>>>>>>>> feature
>> have
>>>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these
>> changes
>>>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in
>> Hadoop
>>>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the
>>>>>>>>>>> 0.18
>>>>>>>>>>> branch.
>>>>>>>>>>>
>>>>>>>>>>> I would like to propose that we apply this same "fix" to
>>>>>>>>>>> sync
>> in
>>>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full
>>>>>>>>>>> semantics of
>>>>>>>>>>> sync, I propose we also disable append (perhaps throw
>>>>>>>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and
>> 0.19.1.
>>>>>>>>>>> We can then take the time needed to fix append properly in
>> 0.21.0.
>>>>>>>>>>>
>>>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
>> fixed.
>>>>>>>>>>>
>>>>>>>>>>> Nigel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>>
RE: Hadoop 0.20.0
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
With the availability of HADOOP-5332 I remove my objection.
> -----Original Message-----
> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> Sent: Wednesday, February 25, 2009 9:32 PM
> To: core-dev@hadoop.apache.org
> Subject: Re: Hadoop 0.20.0
>
> I posted a patch for HADOOP-5332. I am suggesting that this patch be
> applied
> into the 0.19, 0.20 and trunk. This patch switches off "append" by
> default,
> but it can be switched on by setting the config parameter
> dfs.support.append. This does not mean that "append" is bug free in the
> code, it just allows developers to continue testing with append
> functionality till the bugs are fixed.
>
> thanks,
> dhruba
>
> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
> inc.com>wrote:
>
> > +1 for HADOOP-5332. I am in the same position as Brian, as an outside
> > observer. This will help us to move on Hadoop 0.20 which has a lot of
> other
> > features as well that users are asking for.
> >
> > Thanks
> > hemanth
> >
> >
> >
> >> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
> >>
> >>
> >>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
> >>>
> >>> "Whipping out a patch" says nothing about its reliability.
> >>>>
> >>>> i would like some focus from the developer's community to properly
> fix
> >>>> this
> >>>> issue. I am willing to spend as much as time it takes ot get it
> fixed
> >>>> the
> >>>> right way, I but I would like even more constructive engagement
> from
> >>>> more
> >>>> people to get this one right. May I request you to see if you can
> >>>> volunteer
> >>>> to spend some time testing some of this code at scale ?(I have
> access to
> >>>> 10
> >>>> machines only for testing).
> >>>>
> >>>
> >>>
> >> Dhruba, can you define "testing some of this code at scale"? Do you
> simply
> >> need access or folks who can run challenging jobs? Scaring up access
> to the
> >> cluster can be easy, but admin / user time isn't really available.
> >>
> >> Sorry, I can't commit any time/resources to this right now. Perhaps
> some
> >>> hbase folks can. In the meantime, can we make append configurable in
> 0.19.2
> >>> and 0.20.0? I filed
> >>> https://issues.apache.org/jira/browse/HADOOP-5332
> >>>
> >>
> >> As an outside, irrelevant observer, I think this is a really good
> >> compromise. Helps out HBase but also would help prevent rushing.
> >>
> >> Brian
> >>
> >>
> >>>
> >>> Cheers,
> >>> Nige
> >>>
> >>>
> >>>>
> >>>> thanks
> >>>> dhruba
> >>>>
> >>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com>
> >>>> wrote:
> >>>>
> >>>>
> >>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
> >>>>>
> >>>>> Hi Jim,
> >>>>>
> >>>>>>
> >>>>>> I can understand your problem. I can probably whip out a fix for
> >>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
> nice
> >>>>>> if
> >>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
> discuss
> >>>>>> and
> >>>>>> review the patches/fixes.
> >>>>>>
> >>>>>>
> >>>>> "Whipping out a patch" doesn't give me any confidence that this
> feature
> >>>>> will be fixed properly. We're building a file system. Data
> reliability
> >>>>> and
> >>>>> accuracy are absolutely key. We know that this feature has been
> very
> >>>>> lightly tested.
> >>>>>
> >>>>> Nigel: wht is the proposed deadline for 0.20?
> >>>>>
> >>>>>>
> >>>>>>
> >>>>> March 6.
> >>>>>
> >>>>> Nige
> >>>>>
> >>>>>
> >>>>> thanks,
> >>>>>
> >>>>>> dhruba
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
> >>>>>> Jim.Kellerman@microsoft.com> wrote:
> >>>>>>
> >>>>>> --1
> >>>>>>
> >>>>>>>
> >>>>>>> HBase really needs 4379. My testing to date indicates that it
> does
> >>>>>>> work
> >>>>>>> (although I have a bit more testing to do).
> >>>>>>>
> >>>>>>> I was ok with not putting it into 0.19.1 provided it was in
> 0.19.2
> >>>>>>> and
> >>>>>>> 0.20.0.
> >>>>>>>
> >>>>>>> It's a big problem for us now and is hurting our ability to keep
> our
> >>>>>>> community alive. (They will go to Cassandra or something else to
> >>>>>>> ensure
> >>>>>>> reliability).
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>>
> >>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> >>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
> >>>>>>>> To: core-dev@hadoop.apache.org
> >>>>>>>> Subject: Hadoop 0.20.0
> >>>>>>>>
> >>>>>>>> Folks,
> >>>>>>>>
> >>>>>>>> Hadoop 0.19.1 is now available with the file append feature
> >>>>>>>> disabled.
> >>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
> >>>>>>>>
> >>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
> last
> >>>>>>>> few blockers are now almost fixed (should be next week) except
> for
> >>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
> >>>>>>>> implement
> >>>>>>>> file append.
> >>>>>>>>
> >>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
> apply
> >>>>>>>> the
> >>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
> place
> >>>>>>>> to
> >>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
> >>>>>>>>
> >>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Nigel
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Folks,
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
> 0.19
> >>>>>>>>> branch has issues and a 0.19.1 release is needed.
> >>>>>>>>>
> >>>>>>>>> Quality issues in the changes made for the file append feature
> have
> >>>>>>>>> prevented some from deploying Hadoop 0.19. One of these
> changes
> >>>>>>>>> (sync) has now been "fixed" by reducing its semantics in
> Hadoop
> >>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
> >>>>>>>>> branch.
> >>>>>>>>>
> >>>>>>>>> I would like to propose that we apply this same "fix" to sync
> in
> >>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
> >>>>>>>>> sync, I propose we also disable append (perhaps throw
> >>>>>>>>> UnsupportedOperationException from API?). Yes, this would
> >>>>>>>>> unfortunately be an incompatible change between 0.19.0 and
> 0.19.1.
> >>>>>>>>> We can then take the time needed to fix append properly in
> 0.21.0.
> >>>>>>>>>
> >>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
> fixed.
> >>>>>>>>>
> >>>>>>>>> Nigel
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>
> >
Re: Hadoop 0.20.0
Posted by Dhruba Borthakur <dh...@gmail.com>.
I posted a patch for HADOOP-5332. I am suggesting that this patch be applied
into the 0.19, 0.20 and trunk. This patch switches off "append" by default,
but it can be switched on by setting the config parameter
dfs.support.append. This does not mean that "append" is bug free in the
code, it just allows developers to continue testing with append
functionality till the bugs are fixed.
thanks,
dhruba
On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yh...@yahoo-inc.com>wrote:
> +1 for HADOOP-5332. I am in the same position as Brian, as an outside
> observer. This will help us to move on Hadoop 0.20 which has a lot of other
> features as well that users are asking for.
>
> Thanks
> hemanth
>
>
>
>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
>>
>>
>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
>>>
>>> "Whipping out a patch" says nothing about its reliability.
>>>>
>>>> i would like some focus from the developer's community to properly fix
>>>> this
>>>> issue. I am willing to spend as much as time it takes ot get it fixed
>>>> the
>>>> right way, I but I would like even more constructive engagement from
>>>> more
>>>> people to get this one right. May I request you to see if you can
>>>> volunteer
>>>> to spend some time testing some of this code at scale ?(I have access to
>>>> 10
>>>> machines only for testing).
>>>>
>>>
>>>
>> Dhruba, can you define "testing some of this code at scale"? Do you simply
>> need access or folks who can run challenging jobs? Scaring up access to the
>> cluster can be easy, but admin / user time isn't really available.
>>
>> Sorry, I can't commit any time/resources to this right now. Perhaps some
>>> hbase folks can. In the meantime, can we make append configurable in 0.19.2
>>> and 0.20.0? I filed
>>> https://issues.apache.org/jira/browse/HADOOP-5332
>>>
>>
>> As an outside, irrelevant observer, I think this is a really good
>> compromise. Helps out HBase but also would help prevent rushing.
>>
>> Brian
>>
>>
>>>
>>> Cheers,
>>> Nige
>>>
>>>
>>>>
>>>> thanks
>>>> dhruba
>>>>
>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com>
>>>> wrote:
>>>>
>>>>
>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>>>>>
>>>>> Hi Jim,
>>>>>
>>>>>>
>>>>>> I can understand your problem. I can probably whip out a fix for
>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice
>>>>>> if
>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss
>>>>>> and
>>>>>> review the patches/fixes.
>>>>>>
>>>>>>
>>>>> "Whipping out a patch" doesn't give me any confidence that this feature
>>>>> will be fixed properly. We're building a file system. Data reliability
>>>>> and
>>>>> accuracy are absolutely key. We know that this feature has been very
>>>>> lightly tested.
>>>>>
>>>>> Nigel: wht is the proposed deadline for 0.20?
>>>>>
>>>>>>
>>>>>>
>>>>> March 6.
>>>>>
>>>>> Nige
>>>>>
>>>>>
>>>>> thanks,
>>>>>
>>>>>> dhruba
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>>>>>> Jim.Kellerman@microsoft.com> wrote:
>>>>>>
>>>>>> --1
>>>>>>
>>>>>>>
>>>>>>> HBase really needs 4379. My testing to date indicates that it does
>>>>>>> work
>>>>>>> (although I have a bit more testing to do).
>>>>>>>
>>>>>>> I was ok with not putting it into 0.19.1 provided it was in 0.19.2
>>>>>>> and
>>>>>>> 0.20.0.
>>>>>>>
>>>>>>> It's a big problem for us now and is hurting our ability to keep our
>>>>>>> community alive. (They will go to Cassandra or something else to
>>>>>>> ensure
>>>>>>> reliability).
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>>
>>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>>>>>> To: core-dev@hadoop.apache.org
>>>>>>>> Subject: Hadoop 0.20.0
>>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> Hadoop 0.19.1 is now available with the file append feature
>>>>>>>> disabled.
>>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>>>>>
>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
>>>>>>>> few blockers are now almost fixed (should be next week) except for
>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>>>>>> implement
>>>>>>>> file append.
>>>>>>>>
>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
>>>>>>>> the
>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in place
>>>>>>>> to
>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>>>>>
>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Nigel
>>>>>>>>
>>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>>>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>>>>>
>>>>>>>>> Quality issues in the changes made for the file append feature have
>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>>>>>> branch.
>>>>>>>>>
>>>>>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>>>>>> sync, I propose we also disable append (perhaps throw
>>>>>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>>>>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>>>>>>
>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>>>>>>
>>>>>>>>> Nigel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>
>
Re: Hadoop 0.20.0
Posted by Hemanth Yamijala <yh...@yahoo-inc.com>.
+1 for HADOOP-5332. I am in the same position as Brian, as an outside
observer. This will help us to move on Hadoop 0.20 which has a lot of
other features as well that users are asking for.
Thanks
hemanth
>
> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
>
>>
>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
>>
>>> "Whipping out a patch" says nothing about its reliability.
>>>
>>> i would like some focus from the developer's community to properly
>>> fix this
>>> issue. I am willing to spend as much as time it takes ot get it
>>> fixed the
>>> right way, I but I would like even more constructive engagement from
>>> more
>>> people to get this one right. May I request you to see if you can
>>> volunteer
>>> to spend some time testing some of this code at scale ?(I have
>>> access to 10
>>> machines only for testing).
>>
>
> Dhruba, can you define "testing some of this code at scale"? Do you
> simply need access or folks who can run challenging jobs? Scaring up
> access to the cluster can be easy, but admin / user time isn't really
> available.
>
>> Sorry, I can't commit any time/resources to this right now. Perhaps
>> some hbase folks can. In the meantime, can we make append
>> configurable in 0.19.2 and 0.20.0? I filed
>> https://issues.apache.org/jira/browse/HADOOP-5332
>
> As an outside, irrelevant observer, I think this is a really good
> compromise. Helps out HBase but also would help prevent rushing.
>
> Brian
>
>>
>>
>> Cheers,
>> Nige
>>
>>>
>>>
>>> thanks
>>> dhruba
>>>
>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com>
>>> wrote:
>>>
>>>>
>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>>>>
>>>> Hi Jim,
>>>>>
>>>>> I can understand your problem. I can probably whip out a fix for
>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
>>>>> nice if
>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
>>>>> discuss and
>>>>> review the patches/fixes.
>>>>>
>>>>
>>>> "Whipping out a patch" doesn't give me any confidence that this
>>>> feature
>>>> will be fixed properly. We're building a file system. Data
>>>> reliability and
>>>> accuracy are absolutely key. We know that this feature has been very
>>>> lightly tested.
>>>>
>>>> Nigel: wht is the proposed deadline for 0.20?
>>>>>
>>>>
>>>> March 6.
>>>>
>>>> Nige
>>>>
>>>>
>>>> thanks,
>>>>> dhruba
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>>>>> Jim.Kellerman@microsoft.com> wrote:
>>>>>
>>>>> --1
>>>>>>
>>>>>> HBase really needs 4379. My testing to date indicates that it
>>>>>> does work
>>>>>> (although I have a bit more testing to do).
>>>>>>
>>>>>> I was ok with not putting it into 0.19.1 provided it was in
>>>>>> 0.19.2 and
>>>>>> 0.20.0.
>>>>>>
>>>>>> It's a big problem for us now and is hurting our ability to keep our
>>>>>> community alive. (They will go to Cassandra or something else to
>>>>>> ensure
>>>>>> reliability).
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>>>>> To: core-dev@hadoop.apache.org
>>>>>>> Subject: Hadoop 0.20.0
>>>>>>>
>>>>>>> Folks,
>>>>>>>
>>>>>>> Hadoop 0.19.1 is now available with the file append feature
>>>>>>> disabled.
>>>>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>>>>
>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
>>>>>>> few blockers are now almost fixed (should be next week) except for
>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>>>>> implement
>>>>>>> file append.
>>>>>>>
>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
>>>>>>> apply the
>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
>>>>>>> place to
>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>>>>
>>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Nigel
>>>>>>>
>>>>>>>
>>>>>>> Folks,
>>>>>>>>
>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>>>>
>>>>>>>> Quality issues in the changes made for the file append feature
>>>>>>>> have
>>>>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>>>>> branch.
>>>>>>>>
>>>>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>>>>> sync, I propose we also disable append (perhaps throw
>>>>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>>>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>>>>>
>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>>>>>
>>>>>>>> Nigel
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>
Re: Hadoop 0.20.0
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
>
> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
>
>> "Whipping out a patch" says nothing about its reliability.
>>
>> i would like some focus from the developer's community to properly
>> fix this
>> issue. I am willing to spend as much as time it takes ot get it
>> fixed the
>> right way, I but I would like even more constructive engagement
>> from more
>> people to get this one right. May I request you to see if you can
>> volunteer
>> to spend some time testing some of this code at scale ?(I have
>> access to 10
>> machines only for testing).
>
Dhruba, can you define "testing some of this code at scale"? Do you
simply need access or folks who can run challenging jobs? Scaring up
access to the cluster can be easy, but admin / user time isn't really
available.
> Sorry, I can't commit any time/resources to this right now. Perhaps
> some hbase folks can. In the meantime, can we make append
> configurable in 0.19.2 and 0.20.0? I filed
> https://issues.apache.org/jira/browse/HADOOP-5332
As an outside, irrelevant observer, I think this is a really good
compromise. Helps out HBase but also would help prevent rushing.
Brian
>
>
> Cheers,
> Nige
>
>>
>>
>> thanks
>> dhruba
>>
>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com>
>> wrote:
>>
>>>
>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>>>
>>> Hi Jim,
>>>>
>>>> I can understand your problem. I can probably whip out a fix for
>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
>>>> nice if
>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
>>>> discuss and
>>>> review the patches/fixes.
>>>>
>>>
>>> "Whipping out a patch" doesn't give me any confidence that this
>>> feature
>>> will be fixed properly. We're building a file system. Data
>>> reliability and
>>> accuracy are absolutely key. We know that this feature has been
>>> very
>>> lightly tested.
>>>
>>> Nigel: wht is the proposed deadline for 0.20?
>>>>
>>>
>>> March 6.
>>>
>>> Nige
>>>
>>>
>>> thanks,
>>>> dhruba
>>>>
>>>>
>>>>
>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>>>> Jim.Kellerman@microsoft.com> wrote:
>>>>
>>>> --1
>>>>>
>>>>> HBase really needs 4379. My testing to date indicates that it
>>>>> does work
>>>>> (although I have a bit more testing to do).
>>>>>
>>>>> I was ok with not putting it into 0.19.1 provided it was in
>>>>> 0.19.2 and
>>>>> 0.20.0.
>>>>>
>>>>> It's a big problem for us now and is hurting our ability to keep
>>>>> our
>>>>> community alive. (They will go to Cassandra or something else to
>>>>> ensure
>>>>> reliability).
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>>>> To: core-dev@hadoop.apache.org
>>>>>> Subject: Hadoop 0.20.0
>>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Hadoop 0.19.1 is now available with the file append feature
>>>>>> disabled.
>>>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>>>
>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
>>>>>> last
>>>>>> few blockers are now almost fixed (should be next week) except
>>>>>> for
>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>>>> implement
>>>>>> file append.
>>>>>>
>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
>>>>>> apply the
>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
>>>>>> place to
>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>>>
>>>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>>>
>>>>>> Cheers,
>>>>>> Nigel
>>>>>>
>>>>>>
>>>>>> Folks,
>>>>>>>
>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
>>>>>>> 0.19
>>>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>>>
>>>>>>> Quality issues in the changes made for the file append feature
>>>>>>> have
>>>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>>>> branch.
>>>>>>>
>>>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>>>> sync, I propose we also disable append (perhaps throw
>>>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>>>> unfortunately be an incompatible change between 0.19.0 and
>>>>>>> 0.19.1.
>>>>>>> We can then take the time needed to fix append properly in
>>>>>>> 0.21.0.
>>>>>>>
>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are
>>>>>>> fixed.
>>>>>>>
>>>>>>> Nigel
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
Re: Hadoop 0.20.0
Posted by Nigel Daley <nd...@yahoo-inc.com>.
On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:
> "Whipping out a patch" says nothing about its reliability.
>
> i would like some focus from the developer's community to properly
> fix this
> issue. I am willing to spend as much as time it takes ot get it
> fixed the
> right way, I but I would like even more constructive engagement from
> more
> people to get this one right. May I request you to see if you can
> volunteer
> to spend some time testing some of this code at scale ?(I have
> access to 10
> machines only for testing).
Sorry, I can't commit any time/resources to this right now. Perhaps
some hbase folks can. In the meantime, can we make append
configurable in 0.19.2 and 0.20.0? I filed
https://issues.apache.org/jira/browse/HADOOP-5332
Cheers,
Nige
>
>
> thanks
> dhruba
>
> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com>
> wrote:
>
>>
>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>>
>> Hi Jim,
>>>
>>> I can understand your problem. I can probably whip out a fix for
>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
>>> nice if
>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
>>> discuss and
>>> review the patches/fixes.
>>>
>>
>> "Whipping out a patch" doesn't give me any confidence that this
>> feature
>> will be fixed properly. We're building a file system. Data
>> reliability and
>> accuracy are absolutely key. We know that this feature has been very
>> lightly tested.
>>
>> Nigel: wht is the proposed deadline for 0.20?
>>>
>>
>> March 6.
>>
>> Nige
>>
>>
>> thanks,
>>> dhruba
>>>
>>>
>>>
>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>>> Jim.Kellerman@microsoft.com> wrote:
>>>
>>> --1
>>>>
>>>> HBase really needs 4379. My testing to date indicates that it
>>>> does work
>>>> (although I have a bit more testing to do).
>>>>
>>>> I was ok with not putting it into 0.19.1 provided it was in
>>>> 0.19.2 and
>>>> 0.20.0.
>>>>
>>>> It's a big problem for us now and is hurting our ability to keep
>>>> our
>>>> community alive. (They will go to Cassandra or something else to
>>>> ensure
>>>> reliability).
>>>>
>>>>
>>>> -----Original Message-----
>>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>>> To: core-dev@hadoop.apache.org
>>>>> Subject: Hadoop 0.20.0
>>>>>
>>>>> Folks,
>>>>>
>>>>> Hadoop 0.19.1 is now available with the file append feature
>>>>> disabled.
>>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>>
>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
>>>>> last
>>>>> few blockers are now almost fixed (should be next week) except for
>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>>>> implement
>>>>> file append.
>>>>>
>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and
>>>>> apply the
>>>>> same disabling of file append in Hadoop 0.20.0 that we put in
>>>>> place to
>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>>
>>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>>
>>>>> Cheers,
>>>>> Nigel
>>>>>
>>>>>
>>>>> Folks,
>>>>>>
>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
>>>>>> 0.19
>>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>>
>>>>>> Quality issues in the changes made for the file append feature
>>>>>> have
>>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>>> branch.
>>>>>>
>>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>>> sync, I propose we also disable append (perhaps throw
>>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>>> unfortunately be an incompatible change between 0.19.0 and
>>>>>> 0.19.1.
>>>>>> We can then take the time needed to fix append properly in
>>>>>> 0.21.0.
>>>>>>
>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>>>
>>>>>> Nigel
>>>>>>
>>>>>
>>>>
>>>>
>>
Re: Hadoop 0.20.0
Posted by Dhruba Borthakur <dh...@gmail.com>.
"Whipping out a patch" says nothing about its reliability.
i would like some focus from the developer's community to properly fix this
issue. I am willing to spend as much as time it takes ot get it fixed the
right way, I but I would like even more constructive engagement from more
people to get this one right. May I request you to see if you can volunteer
to spend some time testing some of this code at scale ?(I have access to 10
machines only for testing).
thanks
dhruba
On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <nd...@yahoo-inc.com> wrote:
>
> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>
> Hi Jim,
>>
>> I can understand your problem. I can probably whip out a fix for
>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice if
>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss and
>> review the patches/fixes.
>>
>
> "Whipping out a patch" doesn't give me any confidence that this feature
> will be fixed properly. We're building a file system. Data reliability and
> accuracy are absolutely key. We know that this feature has been very
> lightly tested.
>
> Nigel: wht is the proposed deadline for 0.20?
>>
>
> March 6.
>
> Nige
>
>
> thanks,
>> dhruba
>>
>>
>>
>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>> Jim.Kellerman@microsoft.com> wrote:
>>
>> --1
>>>
>>> HBase really needs 4379. My testing to date indicates that it does work
>>> (although I have a bit more testing to do).
>>>
>>> I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
>>> 0.20.0.
>>>
>>> It's a big problem for us now and is hurting our ability to keep our
>>> community alive. (They will go to Cassandra or something else to ensure
>>> reliability).
>>>
>>>
>>> -----Original Message-----
>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>> To: core-dev@hadoop.apache.org
>>>> Subject: Hadoop 0.20.0
>>>>
>>>> Folks,
>>>>
>>>> Hadoop 0.19.1 is now available with the file append feature disabled.
>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>>
>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
>>>> few blockers are now almost fixed (should be next week) except for
>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
>>>> file append.
>>>>
>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
>>>> same disabling of file append in Hadoop 0.20.0 that we put in place to
>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>>
>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>>
>>>> Cheers,
>>>> Nigel
>>>>
>>>>
>>>> Folks,
>>>>>
>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>>> branch has issues and a 0.19.1 release is needed.
>>>>>
>>>>> Quality issues in the changes made for the file append feature have
>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>>> branch.
>>>>>
>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>>> sync, I propose we also disable append (perhaps throw
>>>>> UnsupportedOperationException from API?). Yes, this would
>>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>>
>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>>
>>>>> Nigel
>>>>>
>>>>
>>>
>>>
>
Re: Hadoop 0.20.0
Posted by Nigel Daley <nd...@yahoo-inc.com>.
On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
> Hi Jim,
>
> I can understand your problem. I can probably whip out a fix for
> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
> nice if
> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
> discuss and
> review the patches/fixes.
"Whipping out a patch" doesn't give me any confidence that this
feature will be fixed properly. We're building a file system. Data
reliability and accuracy are absolutely key. We know that this
feature has been very lightly tested.
> Nigel: wht is the proposed deadline for 0.20?
March 6.
Nige
> thanks,
> dhruba
>
>
>
> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
> Jim.Kellerman@microsoft.com> wrote:
>
>> --1
>>
>> HBase really needs 4379. My testing to date indicates that it does
>> work
>> (although I have a bit more testing to do).
>>
>> I was ok with not putting it into 0.19.1 provided it was in 0.19.2
>> and
>> 0.20.0.
>>
>> It's a big problem for us now and is hurting our ability to keep our
>> community alive. (They will go to Cassandra or something else to
>> ensure
>> reliability).
>>
>>
>>> -----Original Message-----
>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>> To: core-dev@hadoop.apache.org
>>> Subject: Hadoop 0.20.0
>>>
>>> Folks,
>>>
>>> Hadoop 0.19.1 is now available with the file append feature
>>> disabled.
>>> It's time to talk about a Hadoop 0.20.0 release.
>>>
>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
>>> few blockers are now almost fixed (should be next week) except for
>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly
>>> implement
>>> file append.
>>>
>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
>>> the
>>> same disabling of file append in Hadoop 0.20.0 that we put in
>>> place to
>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>
>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>
>>> Cheers,
>>> Nigel
>>>
>>>
>>>> Folks,
>>>>
>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>> branch has issues and a 0.19.1 release is needed.
>>>>
>>>> Quality issues in the changes made for the file append feature have
>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
>>>> branch.
>>>>
>>>> I would like to propose that we apply this same "fix" to sync in
>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of
>>>> sync, I propose we also disable append (perhaps throw
>>>> UnsupportedOperationException from API?). Yes, this would
>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>
>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>
>>>> Nigel
>>
>>
Re: Hadoop 0.20.0
Posted by Dhruba Borthakur <dh...@gmail.com>.
Hi Jim,
I can understand your problem. I can probably whip out a fix for
HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice if
somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss and
review the patches/fixes.
Nigel: wht is the proposed deadline for 0.20?
thanks,
dhruba
On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:
> --1
>
> HBase really needs 4379. My testing to date indicates that it does work
> (although I have a bit more testing to do).
>
> I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
> 0.20.0.
>
> It's a big problem for us now and is hurting our ability to keep our
> community alive. (They will go to Cassandra or something else to ensure
> reliability).
>
>
> > -----Original Message-----
> > From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> > Sent: Tuesday, February 24, 2009 4:02 PM
> > To: core-dev@hadoop.apache.org
> > Subject: Hadoop 0.20.0
> >
> > Folks,
> >
> > Hadoop 0.19.1 is now available with the file append feature disabled.
> > It's time to talk about a Hadoop 0.20.0 release.
> >
> > Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
> > few blockers are now almost fixed (should be next week) except for
> > HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
> > file append.
> >
> > *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
> > same disabling of file append in Hadoop 0.20.0 that we put in place to
> > get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
> >
> > I will call a vote for 0.20.0 when blockers are fixed.
> >
> > Cheers,
> > Nigel
> >
> >
> > > Folks,
> > >
> > > Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
> > > branch has issues and a 0.19.1 release is needed.
> > >
> > > Quality issues in the changes made for the file append feature have
> > > prevented some from deploying Hadoop 0.19. One of these changes
> > > (sync) has now been "fixed" by reducing its semantics in Hadoop
> > > 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
> > > branch.
> > >
> > > I would like to propose that we apply this same "fix" to sync in
> > > 0.19.1 and 0.20.0. Since append requires the full semantics of
> > > sync, I propose we also disable append (perhaps throw
> > > UnsupportedOperationException from API?). Yes, this would
> > > unfortunately be an incompatible change between 0.19.0 and 0.19.1.
> > > We can then take the time needed to fix append properly in 0.21.0.
> > >
> > > I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
> > >
> > > Nigel
>
>
Re: Hadoop 0.20.0
Posted by Ted Dunning <te...@gmail.com>.
Indeed, I am one of these community members who are quite possibly going to
find another alternative if hbase doesn't seem more supportable/supported
reasonably soon. Cassandra suffers from the normal facebook open source
syndrome of no documentation, but that begins to look good given the
problems with append.
On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:
> It's a big problem for us now and is hurting our ability to keep our
> community alive. (They will go to Cassandra or something else to ensure
> reliability).
>
--
Ted Dunning, CTO
DeepDyve
RE: Hadoop 0.20.0
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
--1
HBase really needs 4379. My testing to date indicates that it does work
(although I have a bit more testing to do).
I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
0.20.0.
It's a big problem for us now and is hurting our ability to keep our
community alive. (They will go to Cassandra or something else to ensure
reliability).
> -----Original Message-----
> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> Sent: Tuesday, February 24, 2009 4:02 PM
> To: core-dev@hadoop.apache.org
> Subject: Hadoop 0.20.0
>
> Folks,
>
> Hadoop 0.19.1 is now available with the file append feature disabled.
> It's time to talk about a Hadoop 0.20.0 release.
>
> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
> few blockers are now almost fixed (should be next week) except for
> HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
> file append.
>
> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
> same disabling of file append in Hadoop 0.20.0 that we put in place to
> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>
> I will call a vote for 0.20.0 when blockers are fixed.
>
> Cheers,
> Nigel
>
>
> > Folks,
> >
> > Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
> > branch has issues and a 0.19.1 release is needed.
> >
> > Quality issues in the changes made for the file append feature have
> > prevented some from deploying Hadoop 0.19. One of these changes
> > (sync) has now been "fixed" by reducing its semantics in Hadoop
> > 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
> > branch.
> >
> > I would like to propose that we apply this same "fix" to sync in
> > 0.19.1 and 0.20.0. Since append requires the full semantics of
> > sync, I propose we also disable append (perhaps throw
> > UnsupportedOperationException from API?). Yes, this would
> > unfortunately be an incompatible change between 0.19.0 and 0.19.1.
> > We can then take the time needed to fix append properly in 0.21.0.
> >
> > I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
> >
> > Nigel