[jira] [Commented] (ASTERIXDB-2168) Flushed LSM components receive non-increasing LSNs

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ASTERIXDB-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264011#comment-16264011 ] 

ASF subversion and git services commented on ASTERIXDB-2168:
------------------------------------------------------------

Commit 98b9d603e8531139a4668af48092e9c961ee41fb in asterixdb's branch refs/heads/master from [~luochen01]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=98b9d60 ]

[ASTERIXDB-2161] Fix component id manage lifecycle

- user model changes: no
- storage format changes: no
- interface changes: yes. The interface of LMSIOOperationCallback
is changed

Details:
- The current way of management component ids is not correct,
in presence of that multiple partitions sharing the same primary op
tracker. It's possible when a partition is empty/being flushed,
the next flush is scheduled by another partition, which
will disturb the partition. This patch fixes this by
using the same logic of maintaining flushed LSNs to maintain
component id.
- Extend recycle memory component interface to indicate whether it
switches the new component or not.
- Also fixes [ASTERIXDB-2168] to ensure we do not miss latest flushed
LSNs by advancing io callback before finishing flush

Change-Id: Ifc35184c4d431db9af71cab302439e165ee55f54
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2153
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Contrib: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <ba...@gmail.com>


> Flushed LSM components receive non-increasing LSNs
> --------------------------------------------------
>
>                 Key: ASTERIXDB-2168
>                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2168
>             Project: Apache AsterixDB
>          Issue Type: Bug
>          Components: STO - Storage
>            Reporter: Chen Luo
>            Assignee: Chen Luo
>
> Every time when we flush a memory component for a durable dataset, we always force a log record, and use the LSN of the log record as the FLUSH_LSN of the flushed disk component. Since LSNs are strictly increasing, then FLUSH_LSNs of a index on a partition should be strictly increasing as well. However, this properly may not hold in some case.
> One cause (and there may be other causes as well) is that is the ioOperationCallback.afterFinalize is not synchronized on opTracker. When a flush is finished, we set flushRequested to false and advance the readIndex. And in updateLSN, if a component has been requested to flush, we cannot reset the LSN because otherwise we may lose the previous LSN.
> Here is the issue. A memory component has been flushed. Then the next flush is to be scheduled (coming from other partitions), and we try to updateLSN. But this time afterFinalize is not called yet, and we'll skip the newest LSN (which we shouldn't). Then the flush is scheduled, and we call afterFinalize on the previous flushed component (which is already late at this moment).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)