You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bookkeeper.apache.org by Ivan Kelly <iv...@yahoo-inc.com> on 2011/11/23 21:39:54 UTC
[DISCUSS] Release plan for BookKeeper 4.0.0
Hi guys,
This thread is to discuss what will be released with 4.0.0.
First off, in case people are wondering what happened to 3.4.0, what was 3.4.0 is now known as 4.0.0. This is due to API changes, which we felt justified a new major version number.
Now that ZooKeeper 3.4.0 is out, we can start getting a release for BookKeeper 4.0.0 together. The following JIRAs are must haves for the release.
BOOKKEEPER-111 Document bookie recovery feature
BOOKKEEPER-110 Review documentation
BOOKKEEPER-62 Bookie can not start when encountering corrupted records
BOOKKEEPER-112 Bookie Recovery on an open ledger will cause LedgerHandle#close on that ledger to fail
The following are patch available at the moment, so they should be in the release also.
BOOKKEEPER-39 Bookie server failed to restart because of too many ledgers (more than ~50,000 ledgers)
BOOKKEEPER-117 Support multi threads in hedwig cpp client to leverage multi-core hardware
BOOKKEEPER-118 Hedwig client doesn't kill and remove old subscription channel after redirection.
There a final set, which are marked for 4.0.0. I haven't gone through these in detail, but I think they could all be postponed to 4.1.0 if necessary.
BOOKKEEPER-53 race condition of outstandingMsgSet@SubscribeResponseHandler
BOOKKEEPER-72 Fix warnings issued by FindBugs
BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription endlessly when another subscription is already successfully created previously
BOOKKEEPER-56 Race condition of message handler in connection recovery in Hedwig client
Comments/suggestions welcome,
Ivan
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Ivan Kelly <iv...@yahoo-inc.com>.
Replies inline. I spoke with Sijie this morning and reminded me about the change to keep performance constant regardless of number of ledgers. I think this should go in too, since we have the code.
> BOOKKEEPER-39: You had comments there, Ivan, could you give it another
> review?
> BOOKKEEPER-117: It doesn't look strictly necessary, since it is a
> performance issue, but if the patch is reasonable, we could consider
> having it.
> BOOKKEEPER-118: The patch there seems to be preliminary, but the issue
> seems critical, though.
Going to look at them today.
>
>> There a final set, which are marked for 4.0.0. I haven't gone
>> through these in detail, but I think they could all be postponed to
>> 4.1.0 if necessary.
>> BOOKKEEPER-53 race condition of
>> outstandingMsgSet@SubscribeResponseHandler
>> BOOKKEEPER-72 Fix warnings issued by FindBugs
>> BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription
>> endlessly when another subscription is already successfully created
>> previously
>> BOOKKEEPER-56 Race condition of message handler in connection
>> recovery in Hedwig client
>>
>
> Aside from 72, all others seem serious enough. Do you think these
> races are not serious enough?
Im not sure. I have a todo in my list to look at these for the last week, but haven't got around to it yet.
-Ivan
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Ivan Kelly <iv...@yahoo-inc.com>.
Replies inline. I spoke with Sijie this morning and reminded me about the change to keep performance constant regardless of number of ledgers. I think this should go in too, since we have the code.
> BOOKKEEPER-39: You had comments there, Ivan, could you give it another
> review?
> BOOKKEEPER-117: It doesn't look strictly necessary, since it is a
> performance issue, but if the patch is reasonable, we could consider
> having it.
> BOOKKEEPER-118: The patch there seems to be preliminary, but the issue
> seems critical, though.
Going to look at them today.
>
>> There a final set, which are marked for 4.0.0. I haven't gone
>> through these in detail, but I think they could all be postponed to
>> 4.1.0 if necessary.
>> BOOKKEEPER-53 race condition of
>> outstandingMsgSet@SubscribeResponseHandler
>> BOOKKEEPER-72 Fix warnings issued by FindBugs
>> BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription
>> endlessly when another subscription is already successfully created
>> previously
>> BOOKKEEPER-56 Race condition of message handler in connection
>> recovery in Hedwig client
>>
>
> Aside from 72, all others seem serious enough. Do you think these
> races are not serious enough?
Im not sure. I have a todo in my list to look at these for the last week, but haven't got around to it yet.
-Ivan
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
On Nov 23, 2011, at 9:39 PM, Ivan Kelly wrote:
>
> BOOKKEEPER-111 Document bookie recovery feature
> BOOKKEEPER-110 Review documentation
> BOOKKEEPER-62 Bookie can not start when encountering corrupted
> records
> BOOKKEEPER-112 Bookie Recovery on an open ledger will cause
> LedgerHandle#close on that ledger to fail
>
I agree that these should be in.
> The following are patch available at the moment, so they should be
> in the release also.
>
> BOOKKEEPER-39 Bookie server failed to restart because of too many
> ledgers (more than ~50,000 ledgers)
> BOOKKEEPER-117 Support multi threads in hedwig cpp client to
> leverage multi-core hardware
> BOOKKEEPER-118 Hedwig client doesn't kill and remove old
> subscription channel after redirection.
>
I would say that if there is no contention on the patch and the fix is
critical, then we should hold the release until they are fixed. Here
is the status I see for each jira:
BOOKKEEPER-39: You had comments there, Ivan, could you give it another
review?
BOOKKEEPER-117: It doesn't look strictly necessary, since it is a
performance issue, but if the patch is reasonable, we could consider
having it.
BOOKKEEPER-118: The patch there seems to be preliminary, but the issue
seems critical, though.
> There a final set, which are marked for 4.0.0. I haven't gone
> through these in detail, but I think they could all be postponed to
> 4.1.0 if necessary.
> BOOKKEEPER-53 race condition of
> outstandingMsgSet@SubscribeResponseHandler
> BOOKKEEPER-72 Fix warnings issued by FindBugs
> BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription
> endlessly when another subscription is already successfully created
> previously
> BOOKKEEPER-56 Race condition of message handler in connection
> recovery in Hedwig client
>
Aside from 72, all others seem serious enough. Do you think these
races are not serious enough?
-Flavio
flavio
junqueira
research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Ivan Kelly <iv...@yahoo-inc.com>.
> - BOOKKEEPER-66 use IPv4 for builds : currently you cannot build on some
> debian linux boxes with default settings without applying the patch for
> the build, or without skipping tests. (Ivan also mentioned modifying
> shell startup scripts, but the patch solves the build issue at least).
The patch solves the build issue, but if you use the patch on a system which the build issue would occur, you would instantly hit it when you try to run a bookie or hedwig hub. To the user, this would mean that tests would pass cleanly, but actually running the product would fail, which would be very confusing. For this reason, we need the pom.xml and run scripts (bookkeeper-server/bin/bookkeeper & hedwig-server/bin/hedwig) to be fixed at the same time.
>
> - BOOKKEEPER-67 BookieReadWriteTest gets blocked and never finishes :
> this problem also comes up when building from a default debian box, but
> there is no clear fix, apart from tweaking OS settings. I just ran the
> tests on ubuntu 11, and the "too many open files" issue also affects
> other new tests from hedwig.
>
> Note: both of these issues can be fixed by using the right settings on
> the linux os. Maybe it's possible to simply add a build notice.
Im not sure what to do about this. There's no standard way to check the ulimit in java. I'll have a look later and comment on the JIRA.
>
>
> - BOOKKEEPER-37 Ordering of published messages is not preserved when
> doing asynchronous publication : asynchronous publishing is broken.
> Unfortunately, it's a complex issue (much more than I thought initially)
> , requiring quite a few changes, and my understanding is that it could
> be postponed to 4.1 (more focused on communications).
Yes, this will have to wait for 4.1.0. I've created the version in JIRA, so it can be assigned to that release.
-Ivan
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Matthieu Morel <ma...@yahoo-inc.com>.
Hi,
I wonder about these issues: (2 about building, 1 about consistency)
- BOOKKEEPER-66 use IPv4 for builds : currently you cannot build on some
debian linux boxes with default settings without applying the patch for
the build, or without skipping tests. (Ivan also mentioned modifying
shell startup scripts, but the patch solves the build issue at least).
- BOOKKEEPER-67 BookieReadWriteTest gets blocked and never finishes :
this problem also comes up when building from a default debian box, but
there is no clear fix, apart from tweaking OS settings. I just ran the
tests on ubuntu 11, and the "too many open files" issue also affects
other new tests from hedwig.
Note: both of these issues can be fixed by using the right settings on
the linux os. Maybe it's possible to simply add a build notice.
- BOOKKEEPER-37 Ordering of published messages is not preserved when
doing asynchronous publication : asynchronous publishing is broken.
Unfortunately, it's a complex issue (much more than I thought initially)
, requiring quite a few changes, and my understanding is that it could
be postponed to 4.1 (more focused on communications).
Matthieu
On 11/23/11 9:39 PM, Ivan Kelly wrote:
> Hi guys,
>
> This thread is to discuss what will be released with 4.0.0.
>
> First off, in case people are wondering what happened to 3.4.0, what was 3.4.0 is now known as 4.0.0. This is due to API changes, which we felt justified a new major version number.
>
> Now that ZooKeeper 3.4.0 is out, we can start getting a release for BookKeeper 4.0.0 together. The following JIRAs are must haves for the release.
>
> BOOKKEEPER-111 Document bookie recovery feature
> BOOKKEEPER-110 Review documentation
> BOOKKEEPER-62 Bookie can not start when encountering corrupted records
> BOOKKEEPER-112 Bookie Recovery on an open ledger will cause LedgerHandle#close on that ledger to fail
>
> The following are patch available at the moment, so they should be in the release also.
>
> BOOKKEEPER-39 Bookie server failed to restart because of too many ledgers (more than ~50,000 ledgers)
> BOOKKEEPER-117 Support multi threads in hedwig cpp client to leverage multi-core hardware
> BOOKKEEPER-118 Hedwig client doesn't kill and remove old subscription channel after redirection.
>
> There a final set, which are marked for 4.0.0. I haven't gone through these in detail, but I think they could all be postponed to 4.1.0 if necessary.
> BOOKKEEPER-53 race condition of outstandingMsgSet@SubscribeResponseHandler
> BOOKKEEPER-72 Fix warnings issued by FindBugs
> BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription endlessly when another subscription is already successfully created previously
> BOOKKEEPER-56 Race condition of message handler in connection recovery in Hedwig client
>
> Comments/suggestions welcome,
>
> Ivan
Re: [DISCUSS] Release plan for BookKeeper 4.0.0
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
On Nov 23, 2011, at 9:39 PM, Ivan Kelly wrote:
>
> BOOKKEEPER-111 Document bookie recovery feature
> BOOKKEEPER-110 Review documentation
> BOOKKEEPER-62 Bookie can not start when encountering corrupted
> records
> BOOKKEEPER-112 Bookie Recovery on an open ledger will cause
> LedgerHandle#close on that ledger to fail
>
I agree that these should be in.
> The following are patch available at the moment, so they should be
> in the release also.
>
> BOOKKEEPER-39 Bookie server failed to restart because of too many
> ledgers (more than ~50,000 ledgers)
> BOOKKEEPER-117 Support multi threads in hedwig cpp client to
> leverage multi-core hardware
> BOOKKEEPER-118 Hedwig client doesn't kill and remove old
> subscription channel after redirection.
>
I would say that if there is no contention on the patch and the fix is
critical, then we should hold the release until they are fixed. Here
is the status I see for each jira:
BOOKKEEPER-39: You had comments there, Ivan, could you give it another
review?
BOOKKEEPER-117: It doesn't look strictly necessary, since it is a
performance issue, but if the patch is reasonable, we could consider
having it.
BOOKKEEPER-118: The patch there seems to be preliminary, but the issue
seems critical, though.
> There a final set, which are marked for 4.0.0. I haven't gone
> through these in detail, but I think they could all be postponed to
> 4.1.0 if necessary.
> BOOKKEEPER-53 race condition of
> outstandingMsgSet@SubscribeResponseHandler
> BOOKKEEPER-72 Fix warnings issued by FindBugs
> BOOKKEEPER-55 SubscribeReconnectRetryTask might retry subscription
> endlessly when another subscription is already successfully created
> previously
> BOOKKEEPER-56 Race condition of message handler in connection
> recovery in Hedwig client
>
Aside from 72, all others seem serious enough. Do you think these
races are not serious enough?
-Flavio
flavio
junqueira
research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301