You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Konstantin Shvachko <sh...@yahoo-inc.com> on 2009/02/02 20:59:22 UTC
Re: Hadoop 0.19.1
Raghu, thanks for providing the link.
Jim> Are proposing disabling both append and sync?
Jim, this statement is probably too strong.
sync is not disabled per se, you will be able to use it, although
its full semantic is not guaranteed in some failure scenarios.
See more here
https://issues.apache.org/jira/browse/HADOOP-4663#action_12661802
We will have to really disable append (throw UnsupportedOperationException)
because otherwise current solution may lead to a loss of previously existed data.
I agree with Nigel that there is a need for an urgent 0.19.1 release
because a lot of bugs were fixed since 0.18.2 and 0.19.0.
The system is now stable on our clusters with 0.18.3 same fixes went into 0.19.1.
If we try to rush fixing the bugs for append (listed in my comment) we
risk to destabilize the system again, and this is my main concern.
Formally we should not release until a feature is fixed, but I think it
is better to let people use a stable release with limited functionality
rather than having full functionality with a risk of data loss.
Hope this will work for everybody.
--Konstantin
Raghu Angadi wrote:
> Raghu Angadi wrote:
>>> Is it there also that we would find what is involved making append
>>> work in 0.19.1?
>>
>> If one knew what is enough to fix properly, it would be easy. But over
>> last couple of months, there have been many fixes (some of these jiras
>> are listed in one Konstantins HADOOP-4663).
>
> Konstantin's comment I referred to (it was also linked from HADOOP-4663,
> but harder to find).
>
> https://issues.apache.org/jira/browse/HADOOP-5027#action_12668136
>
> Raghu.
>
>> The discussions are still bringing up more cases where the
>> implementation or algorithm should change. But these are improvements
>> for sure. But doubt if I would be ready to call it is 'completely
>> fixed'. It needs time and a lot of testing in large clusters.
>>
>> Personally I am +1 for getting these into 0.19 branch. Most
>> importantly even clusters and application not using append or sync
>> were also affected, thats why extra caution.
>>
>> my 2 cents. hope this does not digress too much from the main topic.
>>
>> Raghu.
>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>> On Fri, Jan 30, 2009 at 2:36 AM, Steve Loughran <st...@apache.org>
>>> wrote:
>>>
>>>> Nigel Daley wrote:
>>>>
>>>>> Folks,
>>>>>
>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>>> branch
>>>>> has issues and a 0.19.1 release is needed.
>>>>>
>>>>> Quality issues in the changes made for the file append feature have
>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>> (sync) has
>>>>> now been "fixed" by reducing its semantics in Hadoop 0.18.3
>>>>> (HADOOP-4997).
>>>>> This was necessary to stabilize the 0.18 branch.
>>>>>
>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>> 0.19.1
>>>>> and 0.20.0. Since append requires the full semantics of sync, I
>>>>> propose we
>>>>> also disable append (perhaps throw UnsupportedOperationException
>>>>> from API?).
>>>>> Yes, this would unfortunately be an incompatible change between
>>>>> 0.19.0 and
>>>>> 0.19.1. We can then take the time needed to fix append properly in
>>>>> 0.21.0.
>>>>>
>>>> I can see some people being unhappy about this, but giving them a
>>>> choice
>>>> between having the filesystem work or not, hopefully they will see the
>>>> merits of the change. And I am +1 to taking time to fix things; fast
>>>> fixes
>>>> often create new problems
>>>>
>>>
>>
>
>
Re: Hadoop 0.19.1
Posted by Dhruba Borthakur <dh...@gmail.com>.
Looks good to me ( I saw this just now). Thanks for rolling 0.19.1.
dhruba
On Wed, Feb 11, 2009 at 12:47 PM, Nigel Daley <nd...@yahoo-inc.com> wrote:
> It sounds like we agree that we should release 0.19.1 with append throwing
> a UnsupportedOperationException and sync getting the same reduced semantics
> as 0.18.3.
>
> I have filed 2 Jira's to get this done:
> http://issues.apache.org/jira/browse/HADOOP-5224
> http://issues.apache.org/jira/browse/HADOOP-5225
>
> When these are committed, I will roll a Hadoop 0.19.1 candidate release.
>
> Nige
>
>
> On Feb 2, 2009, at 2:02 PM, Doug Judd wrote:
>
> Sounds good. I would much rather wait and have fsync() done correctly in
>> 0.20 than get some sort of hacked version in 0.19. I'll create a couple
>> of
>> issues and mark them for 0.20 Thanks.
>>
>> - Doug
>>
>> On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org> wrote:
>>
>> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>>>
>>> What do you recommend? Is there anyway we could get these two issues
>>>
>>>> fixed
>>>> for 0.19.1, or should I file issues for them and get them on the
>>>> schedule
>>>> for 0.19.2?
>>>>
>>>>
>>> Given the outstanding problems and general level of uncertainty, I'd
>>> favor
>>> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on fsync and
>>> append. Let's get them fixed in 0.20 first and then we can debate whether
>>> the rewards of pushing them back into an 0.19.2 would make sense. I'm
>>> pretty
>>> uncomfortable at the moment with how the entire functional complex seems
>>> to
>>> cause a continuous stream of problems.
>>>
>>> -- Owen
>>>
>>>
>
Re: Hadoop 0.19.1
Posted by Nigel Daley <nd...@yahoo-inc.com>.
It sounds like we agree that we should release 0.19.1 with append
throwing a UnsupportedOperationException and sync getting the same
reduced semantics as 0.18.3.
I have filed 2 Jira's to get this done:
http://issues.apache.org/jira/browse/HADOOP-5224
http://issues.apache.org/jira/browse/HADOOP-5225
When these are committed, I will roll a Hadoop 0.19.1 candidate release.
Nige
On Feb 2, 2009, at 2:02 PM, Doug Judd wrote:
> Sounds good. I would much rather wait and have fsync() done
> correctly in
> 0.20 than get some sort of hacked version in 0.19. I'll create a
> couple of
> issues and mark them for 0.20 Thanks.
>
> - Doug
>
> On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org>
> wrote:
>
>> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>>
>> What do you recommend? Is there anyway we could get these two issues
>>> fixed
>>> for 0.19.1, or should I file issues for them and get them on the
>>> schedule
>>> for 0.19.2?
>>>
>>
>> Given the outstanding problems and general level of uncertainty,
>> I'd favor
>> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on
>> fsync and
>> append. Let's get them fixed in 0.20 first and then we can debate
>> whether
>> the rewards of pushing them back into an 0.19.2 would make sense.
>> I'm pretty
>> uncomfortable at the moment with how the entire functional complex
>> seems to
>> cause a continuous stream of problems.
>>
>> -- Owen
>>
Re: Hadoop 0.19.1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Feb 2, 2009, at 6:18 PM, Doug Judd wrote:
> Comments inline ...
>
> On Mon, Feb 2, 2009 at 4:23 PM, Konstantin Shvachko <shv@yahoo-
> inc.com>wrote:
>
> > > What do you recommend?
> >
> > In general. There may be people/organizations, which will not
> compromise
> > on the reduced functionality in favor of the stability, this is
> > understandable.
> > I would propose to create a separate (unofficial experimental)
> branch,
> > which
> > would track changes like HADOOP-4379. The branch may later either
> die when
> > the
> > main stream is fixed or be merged with the trunk if the changes
> proved to
> > be stable.
>
>
> Sure, that sounds reasonable. One thing I would caution against is
> spending
> a lot of time doing incremental patchwork on something that needs a
> ground-up overhaul. I would much rather wait a couple of months
> longer and
> get software that is based on a well thought out design that is
> fundamentally sound. Ultimately that will be the fastest path to
> stability.
>
Agree.
sanjay
>
Re: Hadoop 0.19.1
Posted by Doug Judd <do...@zvents.com>.
Comments inline ...
On Mon, Feb 2, 2009 at 4:23 PM, Konstantin Shvachko <sh...@yahoo-inc.com>wrote:
> > What do you recommend?
>
> In general. There may be people/organizations, which will not compromise
> on the reduced functionality in favor of the stability, this is
> understandable.
> I would propose to create a separate (unofficial experimental) branch,
> which
> would track changes like HADOOP-4379. The branch may later either die when
> the
> main stream is fixed or be merged with the trunk if the changes proved to
> be stable.
Sure, that sounds reasonable. One thing I would caution against is spending
a lot of time doing incremental patchwork on something that needs a
ground-up overhaul. I would much rather wait a couple of months longer and
get software that is based on a well thought out design that is
fundamentally sound. Ultimately that will be the fastest path to stability.
>
> >1. the file length (as returned by getFileStatus) is incorrect
> May be the following work around will be useful.
> If you read from a file you always try to read more data than the length
> reported
> by the name-node. How much more? The size of one block would be enough, or
> even to the next (ceiling) block boundary.
I could certainly implement a workaround, however, from an API standpoint,
the filesystem (IMHO) should always give you a way to obtain the real length
of the file. The semantics of the current getFileStatus() make it difficult
to reason about the state of your filesystem. It basically returns a
"possibly stale" version of the length. I would prefer to wait for an
implementation that gives an accurate answer and spend my time and energy
helping to test that one, rather than spending a bunch of time implementing
a workaround for the current version.
>2. When an application comes up after a crash, it seems to hang for about
> 60
>
> Don't have enough context on that, sorry.
I spoke too soon on this. The reason that HDFS was hanging on lease
recovery was because I was opening the file in append mode to force lease
recovery (at Dhruba's suggestion) so that it would update the NameNode with
the proper length. If I had a method of obtaining the accurate length of
the file, I wouldn't need to do this. Hence, I didn't bother filing an
issue on this.
- Doug
> Thanks,
> --Konstantin
>
> Doug Judd wrote:
>
>> Sounds good. I would much rather wait and have fsync() done correctly in
>> 0.20 than get some sort of hacked version in 0.19. I'll create a couple
>> of
>> issues and mark them for 0.20 Thanks.
>>
>> - Doug
>>
>> On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org> wrote:
>>
>> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>>>
>>> What do you recommend? Is there anyway we could get these two issues
>>>
>>>> fixed
>>>> for 0.19.1, or should I file issues for them and get them on the
>>>> schedule
>>>> for 0.19.2?
>>>>
>>>> Given the outstanding problems and general level of uncertainty, I'd
>>> favor
>>> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on fsync and
>>> append. Let's get them fixed in 0.20 first and then we can debate whether
>>> the rewards of pushing them back into an 0.19.2 would make sense. I'm
>>> pretty
>>> uncomfortable at the moment with how the entire functional complex seems
>>> to
>>> cause a continuous stream of problems.
>>>
>>> -- Owen
>>>
>>>
>>
Re: Hadoop 0.19.1
Posted by Steve Loughran <st...@apache.org>.
Doug Cutting wrote:
> Sanjay Radia wrote:
>> For me the lesson is that large complex projects should be branched.
I currently estimate my merge costs of 20% -one day a week- though a lot
of that is test run time
> If we change the RPC system, or switch DFS client to use RPC instead of
> sockets, etc., we might want to do these in a branch since they'll touch
> a lot of code and will require extensive testing before we release them.
>
> I don't think this is fundamentally a policy issue. We still want to
> demand that things are well tested before we commit them to trunk. The
> append code was committed to trunk prematurely, perhaps since managing
> as a patch was awkward. So this is a praxis issue. For features that
> take a long-time to develop, that we do not want to be forced to
> prematurely commit, a branch is perhaps a better mechanism than a patch.
>
> So I think, if a committer feels a feature requires a branch, then they
> can propose that, and if no one objects, they can create it and maintain
> it. The final commit to trunk is what we should watch most closely,
> since that's the event that corresponds to a commit today. Commits to a
> feature branch should not require reviews, since these are equivalent to
> updating a patch.
>
> Does this sound reasonable?
this sounds good, I would propose my lifecycle changes as an example
-changes bits at the bottom
-adds lots of tests
-needs/encourages review
-lets me do my work under SCM
-let's me hook our work Hudson servers up to it
Putting the stuff in a branch makes it easier for people to work on it,
to check it out and build it etc.
-steve
Re: Hadoop 0.19.1
Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> Commits to a feature branch will send a message to the dev list, like
> any other commit. And when folks commit to a feature branch, they
> should reference the Jira issue id, as in any other commit, so that
> folks browsing Jira can see the commits.
What would be sweet is if commit messages contained a link to the Jira
issue. This would require enhancing the mailer.py script that Apache's
subversion repo uses.
http://subversion.tigris.org/tools_contrib.html#mailer_py
Any Python hacking committers want to take this on?
Doug
Re: Hadoop 0.19.1
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
+1 to branching described above. Good to see experimental branches
considered not-too-evil.
7) If something is not covered in 1-6 common sense should prevail :-)
Raghu.
Doug Cutting wrote:
> Konstantin Shvachko wrote:
>> +1. I agree: no review requirement for feature branches, and 1-5.
>> I would add to this (6) merging a feature branch to an official branch
>> goes through regular patch process, that is, a new jira is created with
>> the patch attachment, which now goes through the review process.
>
> Yes, I forgot (6). But I don't think a new Jira is required. The
> branch is equivalent to a patch. If folks review and +1 it, then it can
> be merged to trunk. An branch-backed issue shouldn't be marked as
> "patch available" until it's ready to be reviewed. Once it has been
> merged to trunk then the branch can be removed.
>
>>> Note that we do *not* want feature branches in external repositories,
>>> since commits there would not generate commit messages to the dev
>>> list nor would they generate links in Jira, etc.
>>
>> Didn't understand this: external to what? What is internal repository?
>
> External to Apache's SVN, i.e., a Git repo on your own server.
>
> Doug
Re: Hadoop 0.19.1
Posted by Doug Cutting <cu...@apache.org>.
Konstantin Shvachko wrote:
> +1. I agree: no review requirement for feature branches, and 1-5.
> I would add to this (6) merging a feature branch to an official branch
> goes through regular patch process, that is, a new jira is created with
> the patch attachment, which now goes through the review process.
Yes, I forgot (6). But I don't think a new Jira is required. The
branch is equivalent to a patch. If folks review and +1 it, then it can
be merged to trunk. An branch-backed issue shouldn't be marked as
"patch available" until it's ready to be reviewed. Once it has been
merged to trunk then the branch can be removed.
>> Note that we do *not* want feature branches in external repositories,
>> since commits there would not generate commit messages to the dev list
>> nor would they generate links in Jira, etc.
>
> Didn't understand this: external to what? What is internal repository?
External to Apache's SVN, i.e., a Git repo on your own server.
Doug
Re: Hadoop 0.19.1
Posted by Steve Loughran <st...@apache.org>.
Konstantin Shvachko wrote:
> Doug Cutting wrote:
>> Commits to a feature branch will send a message to the dev list, like
>> any other commit. And when folks commit to a feature branch, they
>> should reference the Jira issue id, as in any other commit, so that
>> folks browsing Jira can see the commits.
>>
>> When someone starts a feature branch they should note it in the Jira
>> issue, so that folks know to browse the "Subversion Commits" tab to
>> see the patch history. I'd expect this to proceed as follows:
>>
>> 1. A comment proposing that a feature branch be added.
>>
>> 2. A comment by a different committer, endorsing the feature branch,
>> and no comments objecting.
>>
>> 3. A comment stating that the feature branch has been added, what
>> it's url is, and that folks should henceforth use the "Subversion
>> Commits" or "All" tab to track the issue.
>>
>> 4. Committers can commit directly to the feature branch, without
>> reviews. Since committers must have a CLA on file, Apache license is
>> assumed.
>>
>> 5. Non-committers can submit patches against the feature branch to
>> the issue in Jira. These would require the license signoff as usual.
>
> +1. I agree: no review requirement for feature branches, and 1-5.
> I would add to this (6) merging a feature branch to an official branch
> goes through regular patch process, that is, a new jira is created with
> the patch attachment, which now goes through the review process.
I'm not sure about the merge, but it is possible. What is important is
that the branches get reviewed regularly before the big commit day. For
those active branches, that means that we may want some oversight/review
process, and an action plan to deal with unmaintained branches (turn to
jira patch, remove the branch?)
>
>> Everything would still be in public, on the dev list, as now.
>>
>> Note that we do *not* want feature branches in external repositories,
>> since commits there would not generate commit messages to the dev list
>> nor would they generate links in Jira, etc.
>
I'm guessing Doug means Git repos and the like.
Now. apache is starting to set up some Git/SVN sync via github, for
efficient branching . That's the alternate place for me to go with my
code, though I quite like SVN myself.
Also, I am not above using HADOOP- issue tags in the commits to our
sourceforge hosted repository...Jira does have the ability to scan other
repositories if needed.
Re: Hadoop 0.19.1
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
Doug Cutting wrote:
> Commits to a feature branch will send a message to the dev list, like
> any other commit. And when folks commit to a feature branch, they
> should reference the Jira issue id, as in any other commit, so that
> folks browsing Jira can see the commits.
>
> When someone starts a feature branch they should note it in the Jira
> issue, so that folks know to browse the "Subversion Commits" tab to see
> the patch history. I'd expect this to proceed as follows:
>
> 1. A comment proposing that a feature branch be added.
>
> 2. A comment by a different committer, endorsing the feature branch,
> and no comments objecting.
>
> 3. A comment stating that the feature branch has been added, what it's
> url is, and that folks should henceforth use the "Subversion Commits" or
> "All" tab to track the issue.
>
> 4. Committers can commit directly to the feature branch, without
> reviews. Since committers must have a CLA on file, Apache license is
> assumed.
>
> 5. Non-committers can submit patches against the feature branch to the
> issue in Jira. These would require the license signoff as usual.
+1. I agree: no review requirement for feature branches, and 1-5.
I would add to this (6) merging a feature branch to an official branch
goes through regular patch process, that is, a new jira is created with
the patch attachment, which now goes through the review process.
> Everything would still be in public, on the dev list, as now.
>
> Note that we do *not* want feature branches in external repositories,
> since commits there would not generate commit messages to the dev list
> nor would they generate links in Jira, etc.
Didn't understand this: external to what? What is internal repository?
--Konstantin
Re: Hadoop 0.19.1
Posted by Doug Cutting <cu...@apache.org>.
Sanjay Radia wrote:
> On Feb 6, 2009, at 10:35 AM, Doug Cutting wrote:
>> Commits to a
>> feature branch should not require reviews, since these are equivalent to
>> updating a patch.
>
> Agree, but it would be wise for the community to get their feedback to
> the project team
> earlier rather than later when the big patch to the trunk occurs.
> Can Jiras be used to discuss a branch? Or would the project team discuss
> this via another mechanism
> (such as email).
Commits to a feature branch will send a message to the dev list, like
any other commit. And when folks commit to a feature branch, they
should reference the Jira issue id, as in any other commit, so that
folks browsing Jira can see the commits.
When someone starts a feature branch they should note it in the Jira
issue, so that folks know to browse the "Subversion Commits" tab to see
the patch history. I'd expect this to proceed as follows:
1. A comment proposing that a feature branch be added.
2. A comment by a different committer, endorsing the feature branch,
and no comments objecting.
3. A comment stating that the feature branch has been added, what
it's url is, and that folks should henceforth use the "Subversion
Commits" or "All" tab to track the issue.
4. Committers can commit directly to the feature branch, without
reviews. Since committers must have a CLA on file, Apache license is
assumed.
5. Non-committers can submit patches against the feature branch to
the issue in Jira. These would require the license signoff as usual.
Everything would still be in public, on the dev list, as now.
Note that we do *not* want feature branches in external repositories,
since commits there would not generate commit messages to the dev list
nor would they generate links in Jira, etc.
Doug
Re: Hadoop 0.19.1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Feb 6, 2009, at 10:35 AM, Doug Cutting wrote:
> Sanjay Radia wrote:
> > For me the lesson is that large complex projects should be branched.
>
> We already maintain release branches. What's under discussion is the
> maintenance of feature branches. We do this today through patch
> files,
> merging each time they are applied. The proposal is to use a source
> code management tool to manage feature branches, which would be merged
> less often, but using better merge tools.
>
> FWIW, Nutch's transition to mapreduce (in the nascent days of Hadoop)
> was managed as a feature branch. Mike & I worked in a branch for
> about
> six months before merging back to the trunk. For a change of that
> magnitude, we found this to be much simpler than updating a patch.
>
> So we need concrete proposals of features that deserve a branches. In
> retrospect, it seems like append may have been better handled in a
> branch than as a patch, but that's hindsight. What future features do
> we feel demand this?
>
> One possibility is to manage the project split in a branch. We could
> start new repos for the hdfs and mapred sub-projects, but branch the
> core repo. Then changes could continue to be applied to trunk while
> the
> details of the split are worked out. Core changes would be merged to
> the new subprojects and the branch while this is in progress. Once we
> feel the split is solid, we can merge the core branch to trunk and
> open
> the new subprojects for business.
>
> If we change the RPC system, or switch DFS client to use RPC instead
> of
> sockets, etc., we might want to do these in a branch since they'll
> touch
> a lot of code and will require extensive testing before we release
> them.
>
> I don't think this is fundamentally a policy issue. We still want to
> demand that things are well tested before we commit them to trunk.
> The
> append code was committed to trunk prematurely, perhaps since managing
> as a patch was awkward. So this is a praxis issue. For features that
> take a long-time to develop, that we do not want to be forced to
> prematurely commit, a branch is perhaps a better mechanism than a
> patch.
>
> So I think, if a committer feels a feature requires a branch, then
> they
> can propose that, and if no one objects, they can create it and
> maintain
> it. The final commit to trunk is what we should watch most closely,
> since that's the event that corresponds to a commit today. Commits
> to a
> feature branch should not require reviews, since these are
> equivalent to
> updating a patch.
>
> Does this sound reasonable?
>
yes.
> Commits to a
> feature branch should not require reviews, since these are
> equivalent to
> updating a patch.
Agree, but it would be wise for the community to get their feedback
to the project team
earlier rather than later when the big patch to the trunk occurs.
Can Jiras be used to discuss a branch? Or would the project team
discuss this via another mechanism
(such as email).
sanjay
>
>
> Doug
>
Re: Hadoop 0.19.1
Posted by Doug Cutting <cu...@apache.org>.
Sanjay Radia wrote:
> For me the lesson is that large complex projects should be branched.
We already maintain release branches. What's under discussion is the
maintenance of feature branches. We do this today through patch files,
merging each time they are applied. The proposal is to use a source
code management tool to manage feature branches, which would be merged
less often, but using better merge tools.
FWIW, Nutch's transition to mapreduce (in the nascent days of Hadoop)
was managed as a feature branch. Mike & I worked in a branch for about
six months before merging back to the trunk. For a change of that
magnitude, we found this to be much simpler than updating a patch.
So we need concrete proposals of features that deserve a branches. In
retrospect, it seems like append may have been better handled in a
branch than as a patch, but that's hindsight. What future features do
we feel demand this?
One possibility is to manage the project split in a branch. We could
start new repos for the hdfs and mapred sub-projects, but branch the
core repo. Then changes could continue to be applied to trunk while the
details of the split are worked out. Core changes would be merged to
the new subprojects and the branch while this is in progress. Once we
feel the split is solid, we can merge the core branch to trunk and open
the new subprojects for business.
If we change the RPC system, or switch DFS client to use RPC instead of
sockets, etc., we might want to do these in a branch since they'll touch
a lot of code and will require extensive testing before we release them.
I don't think this is fundamentally a policy issue. We still want to
demand that things are well tested before we commit them to trunk. The
append code was committed to trunk prematurely, perhaps since managing
as a patch was awkward. So this is a praxis issue. For features that
take a long-time to develop, that we do not want to be forced to
prematurely commit, a branch is perhaps a better mechanism than a patch.
So I think, if a committer feels a feature requires a branch, then they
can propose that, and if no one objects, they can create it and maintain
it. The final commit to trunk is what we should watch most closely,
since that's the event that corresponds to a commit today. Commits to a
feature branch should not require reviews, since these are equivalent to
updating a patch.
Does this sound reasonable?
Doug
RE: Hadoop 0.19.1
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Sanjay,
> The team at Yahoo had to scramble to recover the lost data,
> put in several emergency patches to deal with
> the append code.
>
This makes it sounds like all of our lost blocks were due to append.
Yes, we did lose many blocks/files in the past few months, but I'd say
it was more of a combination of bugs and not solely from append.
Some that I remember:
HADOOP-4556: in 0.17.2. Nothing to do with append.
HADOOP-4935: Probably more from the new feature of replicating corrupted
blocks (that I requested...)
and append related issues.
I have no opinion about branching, but can we assume that data loss
could happen again and think about a slightly better snapshot?
Current '-upgrade' almost works as long as I have a 'renew!' button
instead of requiring a dfs restart.
Koji
-----Original Message-----
From: Sanjay Radia [mailto:sradia@yahoo-inc.com]
Sent: Thursday, February 05, 2009 4:25 PM
To: core-dev@hadoop.apache.org
Subject: Re: Hadoop 0.19.1
On Feb 4, 2009, at 3:38 AM, Steve Loughran wrote:
> Sanjay Radia wrote:
> >
> > On Feb 2, 2009, at 4:23 PM, Konstantin Shvachko wrote:
> >
> >>
> >> > What do you recommend?
> >>
> >> In general. There may be people/organizations, which will not
> compromise
> >> on the reduced functionality in favor of the stability, this is
> >> understandable.
> >> I would propose to create a separate (unofficial experimental)
> branch,
> >> which
> >> would track changes like HADOOP-4379. The branch may later either
> die
> >> when the
> >> main stream is fixed or be merged with the trunk if the changes
> proved
> >> to be stable.
> >>
> >
> >
> > This is very a interesting suggestion.
> > Many in the team have come to the conclusion that complex
> projects like
> > append should be done on a separate branch in the first place and
> > integrated with trunk when the project is stable.
> >
>
> There's a lot to be said for branching; I'm also looking at git so I
> can
> do my service lifecycle stuff under SCM properly.
>
> but the cost of merging can be high. I'd estimate 1 morning/week is
> spent updating my local SVN and then seeing that everything still
> works.
> If hudson could both test the branches and test any merged branches,
> life would be better
>
I agree on the cost of merging.
When a project is branched, after a while one can spend as much as
30% of cycles merging
in changes.
But when a system is used in production to store data we cannot afford
to have users loose their data.
The team at Yahoo had to scramble to recover the lost data, put in
several emergency patches to deal with
the append code.
I am all for extending hudson testing for branches, but hudson
testing, while helpful, will not be sufficient for big
projects because hudson does not have a comprehensive set of tests.
Each new release is tested significantly beyond the hudson tests.
For me the lesson is that large complex projects should be branched.
(This is how commercial software products are engineered).
There will increased cost to the project team, but over all, the
community will have more solid releases and the total cost to the
community in delivering the techology will be smaller.
sanjay
>
>
> The other problem is incompatible branches: the more branches you have
> live, the higher the merge cost.
>
> That said, Git promises wonderful things, and we ought to be able to
> set
> up Apache support for git for people wanting to do their own branches
> -svn would still be the official SCM tool
>
Re: Hadoop 0.19.1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Feb 4, 2009, at 3:38 AM, Steve Loughran wrote:
> Sanjay Radia wrote:
> >
> > On Feb 2, 2009, at 4:23 PM, Konstantin Shvachko wrote:
> >
> >>
> >> > What do you recommend?
> >>
> >> In general. There may be people/organizations, which will not
> compromise
> >> on the reduced functionality in favor of the stability, this is
> >> understandable.
> >> I would propose to create a separate (unofficial experimental)
> branch,
> >> which
> >> would track changes like HADOOP-4379. The branch may later either
> die
> >> when the
> >> main stream is fixed or be merged with the trunk if the changes
> proved
> >> to be stable.
> >>
> >
> >
> > This is very a interesting suggestion.
> > Many in the team have come to the conclusion that complex
> projects like
> > append should be done on a separate branch in the first place and
> > integrated with trunk when the project is stable.
> >
>
> There's a lot to be said for branching; I'm also looking at git so I
> can
> do my service lifecycle stuff under SCM properly.
>
> but the cost of merging can be high. I'd estimate 1 morning/week is
> spent updating my local SVN and then seeing that everything still
> works.
> If hudson could both test the branches and test any merged branches,
> life would be better
>
I agree on the cost of merging.
When a project is branched, after a while one can spend as much as
30% of cycles merging
in changes.
But when a system is used in production to store data we cannot afford
to have users loose their data.
The team at Yahoo had to scramble to recover the lost data, put in
several emergency patches to deal with
the append code.
I am all for extending hudson testing for branches, but hudson
testing, while helpful, will not be sufficient for big
projects because hudson does not have a comprehensive set of tests.
Each new release is tested significantly beyond the hudson tests.
For me the lesson is that large complex projects should be branched.
(This is how commercial software products are engineered).
There will increased cost to the project team, but over all, the
community will have more solid releases and the total cost to the
community in delivering the techology will be smaller.
sanjay
>
>
> The other problem is incompatible branches: the more branches you have
> live, the higher the merge cost.
>
> That said, Git promises wonderful things, and we ought to be able to
> set
> up Apache support for git for people wanting to do their own branches
> -svn would still be the official SCM tool
>
Re: Hadoop 0.19.1
Posted by Steve Loughran <st...@apache.org>.
Sanjay Radia wrote:
>
> On Feb 2, 2009, at 4:23 PM, Konstantin Shvachko wrote:
>
>>
>> > What do you recommend?
>>
>> In general. There may be people/organizations, which will not compromise
>> on the reduced functionality in favor of the stability, this is
>> understandable.
>> I would propose to create a separate (unofficial experimental) branch,
>> which
>> would track changes like HADOOP-4379. The branch may later either die
>> when the
>> main stream is fixed or be merged with the trunk if the changes proved
>> to be stable.
>>
>
>
> This is very a interesting suggestion.
> Many in the team have come to the conclusion that complex projects like
> append should be done on a separate branch in the first place and
> integrated with trunk when the project is stable.
>
There's a lot to be said for branching; I'm also looking at git so I can
do my service lifecycle stuff under SCM properly.
but the cost of merging can be high. I'd estimate 1 morning/week is
spent updating my local SVN and then seeing that everything still works.
If hudson could both test the branches and test any merged branches,
life would be better
The other problem is incompatible branches: the more branches you have
live, the higher the merge cost.
That said, Git promises wonderful things, and we ought to be able to set
up Apache support for git for people wanting to do their own branches
-svn would still be the official SCM tool
Re: Hadoop 0.19.1
Posted by stack <st...@duboce.net>.
On Tue, Feb 3, 2009 at 7:02 PM, Sanjay Radia <sr...@yahoo-inc.com> wrote:
>
> Many in the team have come to the conclusion that complex projects like
> append should be done on a separate branch in the first place and integrated
> with trunk when the project is stable.
>
How do we determine when a feature on the branch is 'stable'? Is there a
test suite to run beyond hudson's running of unit tests?
St.Ack
Re: Hadoop 0.19.1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Feb 2, 2009, at 4:23 PM, Konstantin Shvachko wrote:
>
> > What do you recommend?
>
> In general. There may be people/organizations, which will not
> compromise
> on the reduced functionality in favor of the stability, this is
> understandable.
> I would propose to create a separate (unofficial experimental)
> branch, which
> would track changes like HADOOP-4379. The branch may later either
> die when the
> main stream is fixed or be merged with the trunk if the changes
> proved to be stable.
>
This is very a interesting suggestion.
Many in the team have come to the conclusion that complex projects
like append should be done on a separate branch in the first place and
integrated with trunk when the project is stable.
sanjay
>
>
> >1. the file length (as returned by getFileStatus) is incorrect
>
> May be the following work around will be useful.
> If you read from a file you always try to read more data than the
> length reported
> by the name-node. How much more? The size of one block would be
> enough, or
> even to the next (ceiling) block boundary.
>
> >2. When an application comes up after a crash, it seems to hang
> for about 60
>
> Don't have enough context on that, sorry.
>
> Thanks,
> --Konstantin
>
> Doug Judd wrote:
> > Sounds good. I would much rather wait and have fsync() done
> correctly in
> > 0.20 than get some sort of hacked version in 0.19. I'll create a
> couple of
> > issues and mark them for 0.20 Thanks.
> >
> > - Doug
> >
> > On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org>
> wrote:
> >
> >> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
> >>
> >> What do you recommend? Is there anyway we could get these two
> issues
> >>> fixed
> >>> for 0.19.1, or should I file issues for them and get them on the
> schedule
> >>> for 0.19.2?
> >>>
> >> Given the outstanding problems and general level of uncertainty,
> I'd favor
> >> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on
> fsync and
> >> append. Let's get them fixed in 0.20 first and then we can debate
> whether
> >> the rewards of pushing them back into an 0.19.2 would make sense.
> I'm pretty
> >> uncomfortable at the moment with how the entire functional
> complex seems to
> >> cause a continuous stream of problems.
> >>
> >> -- Owen
> >>
> >
>
Re: Hadoop 0.19.1
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
> What do you recommend?
In general. There may be people/organizations, which will not compromise
on the reduced functionality in favor of the stability, this is understandable.
I would propose to create a separate (unofficial experimental) branch, which
would track changes like HADOOP-4379. The branch may later either die when the
main stream is fixed or be merged with the trunk if the changes proved to be stable.
>1. the file length (as returned by getFileStatus) is incorrect
May be the following work around will be useful.
If you read from a file you always try to read more data than the length reported
by the name-node. How much more? The size of one block would be enough, or
even to the next (ceiling) block boundary.
>2. When an application comes up after a crash, it seems to hang for about 60
Don't have enough context on that, sorry.
Thanks,
--Konstantin
Doug Judd wrote:
> Sounds good. I would much rather wait and have fsync() done correctly in
> 0.20 than get some sort of hacked version in 0.19. I'll create a couple of
> issues and mark them for 0.20 Thanks.
>
> - Doug
>
> On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org> wrote:
>
>> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>>
>> What do you recommend? Is there anyway we could get these two issues
>>> fixed
>>> for 0.19.1, or should I file issues for them and get them on the schedule
>>> for 0.19.2?
>>>
>> Given the outstanding problems and general level of uncertainty, I'd favor
>> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on fsync and
>> append. Let's get them fixed in 0.20 first and then we can debate whether
>> the rewards of pushing them back into an 0.19.2 would make sense. I'm pretty
>> uncomfortable at the moment with how the entire functional complex seems to
>> cause a continuous stream of problems.
>>
>> -- Owen
>>
>
Re: Hadoop 0.19.1
Posted by Doug Judd <do...@zvents.com>.
Sounds good. I would much rather wait and have fsync() done correctly in
0.20 than get some sort of hacked version in 0.19. I'll create a couple of
issues and mark them for 0.20 Thanks.
- Doug
On Mon, Feb 2, 2009 at 1:51 PM, Owen O'Malley <om...@apache.org> wrote:
> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>
> What do you recommend? Is there anyway we could get these two issues
>> fixed
>> for 0.19.1, or should I file issues for them and get them on the schedule
>> for 0.19.2?
>>
>
> Given the outstanding problems and general level of uncertainty, I'd favor
> releasing a 0.19.1 with the equivalent of the 0.18.3 disable on fsync and
> append. Let's get them fixed in 0.20 first and then we can debate whether
> the rewards of pushing them back into an 0.19.2 would make sense. I'm pretty
> uncomfortable at the moment with how the entire functional complex seems to
> cause a continuous stream of problems.
>
> -- Owen
>
RE: Hadoop 0.19.1
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
HBase could live with sync() in 0.20.0 (and preferably 0.19.2),
but 0.20.0 is an absolute blocker.
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: Owen O'Malley [mailto:owen.omalley@gmail.com] On Behalf Of Owen
> O'Malley
> Sent: Monday, February 02, 2009 1:51 PM
> To: core-dev@hadoop.apache.org
> Subject: Re: Hadoop 0.19.1
>
> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>
> > What do you recommend? Is there anyway we could get these two
> > issues fixed
> > for 0.19.1, or should I file issues for them and get them on the
> > schedule
> > for 0.19.2?
>
> Given the outstanding problems and general level of uncertainty, I'd
> favor releasing a 0.19.1 with the equivalent of the 0.18.3 disable on
> fsync and append. Let's get them fixed in 0.20 first and then we can
> debate whether the rewards of pushing them back into an 0.19.2 would
> make sense. I'm pretty uncomfortable at the moment with how the entire
> functional complex seems to cause a continuous stream of problems.
>
> -- Owen
Re: Hadoop 0.19.1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Feb 2, 2009, at 1:51 PM, Owen O'Malley wrote:
> On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
>
> > What do you recommend? Is there anyway we could get these two
> > issues fixed
> > for 0.19.1, or should I file issues for them and get them on the
> > schedule
> > for 0.19.2?
>
> Given the outstanding problems and general level of uncertainty, I'd
> favor releasing a 0.19.1 with the equivalent of the 0.18.3 disable on
> fsync and append. Let's get them fixed in 0.20 first and then we can
> debate whether the rewards of pushing them back into an 0.19.2 would
> make sense. I'm pretty uncomfortable at the moment with how the entire
> functional complex seems to cause a continuous stream of problems.
>
> -- Owen
>
+1
Append/sync code has been very problematic and bug fixes have not
always fixed things completely.
Hence getting a quick release of 0.19.1 that deal with the data loss
(ie 0.18.3 functionality) is important.
sanjay
Re: Hadoop 0.19.1
Posted by Owen O'Malley <om...@apache.org>.
On Feb 2, 2009, at 12:51 PM, Doug Judd wrote:
> What do you recommend? Is there anyway we could get these two
> issues fixed
> for 0.19.1, or should I file issues for them and get them on the
> schedule
> for 0.19.2?
Given the outstanding problems and general level of uncertainty, I'd
favor releasing a 0.19.1 with the equivalent of the 0.18.3 disable on
fsync and append. Let's get them fixed in 0.20 first and then we can
debate whether the rewards of pushing them back into an 0.19.2 would
make sense. I'm pretty uncomfortable at the moment with how the entire
functional complex seems to cause a continuous stream of problems.
-- Owen
Re: Hadoop 0.19.1
Posted by Doug Judd <do...@zvents.com>.
Hi Konstantin,
We are also heavy users of fsync(). I've been working with Dhruba on
Hadoop-4379 <https://issues.apache.org/jira/browse/HADOOP-4379>. His most
recent patch appears to work for Jim's situation. However, there are still
a couple of problems that need to be resolved before we can start heavily
using it:
1. After application crash/restart, the file length (as returned by
getFileStatus) is incorrect since the length at the namenode is stale.
Ideally getFileStatus() would return the accurate file length by fetching
the size of the last block from the primary datanode. If that's not
feasible, there should be some other way to obtain the actual file length.
2. When an application comes up after a crash, it seems to hang for about 60
seconds waiting for lease recovery. Our database cannot go offline for a
whole minute doing nothing. In our case, when we come up after a crash and
try to re-open the log file, we know for certain that we are the exclusive
owner of that file. There should be a way to tell the system to forcibly
take over the lease and recover immediately
What do you recommend? Is there anyway we could get these two issues fixed
for 0.19.1, or should I file issues for them and get them on the schedule
for 0.19.2?
- Doug
On Mon, Feb 2, 2009 at 11:59 AM, Konstantin Shvachko <sh...@yahoo-inc.com>wrote:
> Raghu, thanks for providing the link.
>
> Jim> Are proposing disabling both append and sync?
>
> Jim, this statement is probably too strong.
> sync is not disabled per se, you will be able to use it, although
> its full semantic is not guaranteed in some failure scenarios.
> See more here
> https://issues.apache.org/jira/browse/HADOOP-4663#action_12661802
>
> We will have to really disable append (throw UnsupportedOperationException)
> because otherwise current solution may lead to a loss of previously existed
> data.
>
> I agree with Nigel that there is a need for an urgent 0.19.1 release
> because a lot of bugs were fixed since 0.18.2 and 0.19.0.
> The system is now stable on our clusters with 0.18.3 same fixes went into
> 0.19.1.
>
> If we try to rush fixing the bugs for append (listed in my comment) we
> risk to destabilize the system again, and this is my main concern.
>
> Formally we should not release until a feature is fixed, but I think it
> is better to let people use a stable release with limited functionality
> rather than having full functionality with a risk of data loss.
>
> Hope this will work for everybody.
> --Konstantin
>
>
> Raghu Angadi wrote:
>
>> Raghu Angadi wrote:
>>
>>> Is it there also that we would find what is involved making append
>>>> work in 0.19.1?
>>>>
>>>
>>> If one knew what is enough to fix properly, it would be easy. But over
>>> last couple of months, there have been many fixes (some of these jiras are
>>> listed in one Konstantins HADOOP-4663).
>>>
>>
>> Konstantin's comment I referred to (it was also linked from HADOOP-4663,
>> but harder to find).
>>
>> https://issues.apache.org/jira/browse/HADOOP-5027#action_12668136
>>
>> Raghu.
>>
>> The discussions are still bringing up more cases where the implementation
>>> or algorithm should change. But these are improvements for sure. But doubt
>>> if I would be ready to call it is 'completely fixed'. It needs time and a
>>> lot of testing in large clusters.
>>>
>>> Personally I am +1 for getting these into 0.19 branch. Most importantly
>>> even clusters and application not using append or sync were also affected,
>>> thats why extra caution.
>>>
>>> my 2 cents. hope this does not digress too much from the main topic.
>>>
>>> Raghu.
>>>
>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Fri, Jan 30, 2009 at 2:36 AM, Steve Loughran <st...@apache.org>
>>>> wrote:
>>>>
>>>> Nigel Daley wrote:
>>>>>
>>>>> Folks,
>>>>>>
>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
>>>>>> branch
>>>>>> has issues and a 0.19.1 release is needed.
>>>>>>
>>>>>> Quality issues in the changes made for the file append feature have
>>>>>> prevented some from deploying Hadoop 0.19. One of these changes
>>>>>> (sync) has
>>>>>> now been "fixed" by reducing its semantics in Hadoop 0.18.3
>>>>>> (HADOOP-4997).
>>>>>> This was necessary to stabilize the 0.18 branch.
>>>>>>
>>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>>> 0.19.1
>>>>>> and 0.20.0. Since append requires the full semantics of sync, I
>>>>>> propose we
>>>>>> also disable append (perhaps throw UnsupportedOperationException from
>>>>>> API?).
>>>>>> Yes, this would unfortunately be an incompatible change between
>>>>>> 0.19.0 and
>>>>>> 0.19.1. We can then take the time needed to fix append properly in
>>>>>> 0.21.0.
>>>>>>
>>>>>> I can see some people being unhappy about this, but giving them a
>>>>> choice
>>>>> between having the filesystem work or not, hopefully they will see the
>>>>> merits of the change. And I am +1 to taking time to fix things; fast
>>>>> fixes
>>>>> often create new problems
>>>>>
>>>>>
>>>>
>>>
>>
>>