You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by sijie <gi...@git.apache.org> on 2017/06/02 20:39:36 UTC
[GitHub] bookkeeper pull request #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
GitHub user sijie opened a pull request:
https://github.com/apache/bookkeeper/pull/180
BOOKKEEPER-1093: Piggyback LAC on ReadResponse
This change is based #178 - (you can review git sha 40ca8c2)
bookkeeper: LAC piggyback at read response
- bookie server changes
* cache maximum lac in file info
* provide getLastAddConfirmed & setLastAddConfirmed in ledger storage
* addEntry will set its lac thru setLastAddConfirmed
* readEntry will read latest lac and piggyback
- client change
* check whether the response has lac piggybacked, if there is lac update lac in its corresponding ledger handle.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sijie/bookkeeper piggyback_lac
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bookkeeper/pull/180.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #180
----
commit 3db0b84570dec166855c70105a5cbf6f2ec14ea2
Author: Sijie Guo <si...@twitter.com>
Date: 2014-01-21T00:42:28Z
bookkeeper recovery improvement (part-1): refactor PendingReadOp
this change is the first part of improving bookkeeper recovery. it is basically a refactor change, which:
- abstract an interface for LedgerEntryRequest in PendingReadOp
- rename current implementation to SequenceReadRequest, which read the entry in the sequence of quorum.
RB_ID=266137
commit 80ffc6ca86b5e8a63dc3b9baf846ad139d7beea1
Author: Sijie Guo <si...@apache.org>
Date: 2017-06-01T19:11:40Z
Address conflicts
commit f0fb89cbdfed6c6b0d519a25ea2ea67dcd72834e
Author: Sijie Guo <si...@twitter.com>
Date: 2014-01-21T00:49:52Z
bookkeeper recovery improvement (part-2): add a parallel reading request in PendingReadOp
- add a parallel reading request in PendingReadOp
- allow PendingReadOp to configure whether to do parallel reading or not
- add flag in ClientConfiguration to allow configuring whether to do parallel reading in LedgerRecoveryOp or not.
RB_ID=266139
commit db3e98ba9250b622db80cac8dd6dbd2761a32381
Author: Sijie Guo <si...@apache.org>
Date: 2017-06-01T20:00:01Z
Address conflicts
commit 8b8a3c8db86a11a30efb14d04b33bfcd461a6f05
Author: Sijie Guo <si...@twitter.com>
Date: 2014-01-21T00:55:30Z
bookkeeper recovery improvement (part-3): add a ReadEntryListener to callback on individual request.
- add read entry listener which allow doing batch read, but callback on individual entries in sequence. so in recovery op, we could issue batch reads, then on each individual callback do add entry and stop when received NoSuchEntry.
RB_ID=266143
commit 455afdad38103db978638a9e3f8774623efbb82a
Author: Sijie Guo <si...@twitter.com>
Date: 2014-01-21T01:00:30Z
bookkeeper recovery improvement (part-4): allow batch reading in ledger recovery
- enable batch read in ledger recovery, so we could parallel reading to improve recovery time.
RB_ID=266145
commit d9b9d98518d8ee0b1951ad2eee8ce28820a4c652
Author: Sijie Guo <si...@apache.org>
Date: 2017-06-01T22:04:44Z
Address conflicts in merge
commit 40ca8c2116ae072d459af5d0eca47d6d6a3846ec
Author: Sijie Guo <si...@apache.org>
Date: 2017-06-02T20:36:03Z
bookkeeper: LAC piggyback at read response
- bookie server changes
* cache maximum lac in file info
* provide getLastAddConfirmed & setLastAddConfirmed in ledger storage
* addEntry will set its lac thru setLastAddConfirmed
* readEntry will read latest lac and piggyback
- client change
* check whether the response has lac piggybacked, if there is lac update lac in its corresponding ledger handle.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bookkeeper pull request #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bookkeeper/pull/180
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bookkeeper issue #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
Posted by sijie <gi...@git.apache.org>.
Github user sijie commented on the issue:
https://github.com/apache/bookkeeper/pull/180
you still need to call "readLastAddConfirmed". This change is not "long poll", it is "piggy-back lac". It is an optimization on reducing the times of calling readLastAddConfirmed. In a tailing case, most of the time you should call getLastAddConfirmed, you keep reading until your current read entry is moving beyond the value that #getLastAddConfirmed, which means you caught up with the writer then you switch to call #readLastAddConfirmed. since lac is piggy-back, so the value returned by #readLastConfirmed is advancing, hence you don't need to call #readLastAddConfirmed that often.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bookkeeper issue #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
Posted by sijie <gi...@git.apache.org>.
Github user sijie commented on the issue:
https://github.com/apache/bookkeeper/pull/180
@merlimat ??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bookkeeper issue #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
Posted by eolivelli <gi...@git.apache.org>.
Github user eolivelli commented on the issue:
https://github.com/apache/bookkeeper/pull/180
@sijie I did a more deep review and it is all great for me.
Do you think it would be useful to add a test for readUnconfirmedEntries ? Actually the implementation is shared with readEntries so maybe we can live without such new test
b.q. Just to understand better
How are you using this feature to improve speed of "tailing reads" ? I see that the LAC is advanced without calling readLastAddConfirmed but there is no guarantee that it will actually advance so sometimes you will need to call readLastAddConfirmed anyway, isn't it ?.
How can you distinguish the case of dead writer/no more data/closed ledger ? by catching exceptions ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bookkeeper issue #180: BOOKKEEPER-1093: Piggyback LAC on ReadResponse
Posted by sijie <gi...@git.apache.org>.
Github user sijie commented on the issue:
https://github.com/apache/bookkeeper/pull/180
@merlimat review?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---