Re: [DISCUSS] markdelete position not moving forward when skipping lost entries

[PengHui Li <pe...@apache.org>]

Hi lordcheng,

Sorry for the late reply.

IMO, from the Pulsar side, we remove a ledger from a topic, not the entry.
After I thought about it, I think we should fix the issue from the
BookKeeper side.
The BookKeeper also don't support delete entries, so if the ledger still
there but
bookkeeper loses some data of this ledger, it should be a BUG in
bookkeeper,
it's not an expected behavior, Pulsar needs to be built on bookkeeper
integrity.

To skip the whole ledger is more reasonable, because BookKeeper support
delete ledgers, for some reason such like cleanup data from bookies or
other cases,
users might need to delete some ledgers, which means the risk of losing
data is
already acceptable, so the broker side can skip the whole ledger.

Thanks,
Penghui

On Wed, Mar 16, 2022 at 10:50 AM lordcheng10 <15...@qq.com.invalid>
wrote:

> Here I add, the scene of losing data at that time:
> 1. For the bookkeeper cluster, we turned off the journal writing function;
> 2. Then during the upgrade, after executing the stop command several
> times, the bookkeeper process does not exit, we will kill -9;
> 3. Kill -9 will cause the data in the cache to be too late to flush the
> disk;
> 4. We have two copies, if kill -9 twice in a row, it may lead to the loss
> of all copy data of an entry;