You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@activemq.apache.org by shoukunhuai <gi...@git.apache.org> on 2018/01/03 09:41:36 UTC

[GitHub] activemq-artemis pull request #1747: [WIP] NO-JIRA DISCUSS Replica use opera...

GitHub user shoukunhuai opened a pull request:

    https://github.com/apache/activemq-artemis/pull/1747

    [WIP] NO-JIRA DISCUSS Replica use operationcontext

    This PR is used to discuss using `OperationContext` on backup side.
    
    The original implementation of  `ReplicationEndpoint` sends response to live after journal operation is submitted to executor in `JournalImpl`.
    
    By introduce operation context, we send response after journal operation is complete.
    
    Since we never sync on backup side, time buffer must be disabled, or the responses will not be sent until time buffer is full.
    
    Any suggestion is appreciated.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shoukunhuai/activemq-artemis replica-use-operationcontext

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/activemq-artemis/pull/1747.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1747
    
----
commit b071cf9f6f566276bcfd810bbc622c743b73ab9f
Author: shoukun <sh...@...>
Date:   2017-12-28T03:33:13Z

    Use operation context on backup side
    
    With operaton context we can send response after
    journal is wrote to journal file. Since we always use
    noSync, the change to journal file is not forced.

commit af85f3e2513601a5f5255969664ba40c5980d4d2
Author: shoukun <sh...@...>
Date:   2018-01-03T09:21:01Z

    Disable time buffer on backup side
    
    Since we never sync on backup side, time buffer must be disabled.

----


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by andytaylor <gi...@git.apache.org>.

Github user andytaylor commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    Yes
    
    On 11 Jan 2018 19:52, "Justin Bertram" <no...@github.com> wrote:
    
    > Given the discussion here my understanding is that this PR should be
    > closed. Are we in agreement?
    >
    > —
    > You are receiving this because you commented.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/activemq-artemis/pull/1747#issuecomment-357042102>,
    > or mute the thread
    > <https://github.com/notifications/unsubscribe-auth/AAt5shPmjpt8xsdmSXB260kjFcmLefG4ks5tJmZsgaJpZM4RRg38>
    > .
    >



---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by andytaylor <gi...@git.apache.org>.

Github user andytaylor commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    There is no point in waiting until its written to disc, it just adds extra latency and gives you no extra guarantees. If the backup crashes when the live is still up then it would start replicating from scratch again anyway. If it crashes during the window after the live has crashed but before the backup has finished synching then you have lost HA anyway, however you still have data integrity at the original live broker 


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by yourcaptain <gi...@git.apache.org>.

Github user yourcaptain commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    Maybe it is quite important for shoukunhuai that, when server response to client side 'OK', it must be truely that no data will be lost (with the topology of only 1 master- 1 slave, no extra salves).  
    
    May be under 1master-1slave topology striction,  he can torrance a bit perfomance lost, but cares the garantees to clients.



---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by andytaylor <gi...@git.apache.org>.

Github user andytaylor commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    Michael is correct, that's what I meant. Let me explain how this works. 
    
    Every record that is written to the journal is also sent to the backup. 
    Once the backup receives the packet it responds with an ack and in parallel writes it to disc.
    When the live crashes the back up fsyncs and then starts.
    
    There is a small window between when the live crashes and when the fsync occurs that the backup may crash, but this is unlikely to happen and you stil have data consistency as the journal is on the live.
    
    The caveat here is if the disc on the live was destroyed, but again the window is so small there are more likely scenarios that we could improve from a crash pov.
    
    @shoukunhuai  Why don't you explain your use case, not in an implementation way but something like 'when x happens I want to avoid y'.


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by shoukunhuai <gi...@git.apache.org>.

Github user shoukunhuai commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    @jbertram sure, i will close it, and thanks all.


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by shoukunhuai <gi...@git.apache.org>.

Github user shoukunhuai commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    @michaelandrepearce we want the guarantee that when we response to client, any journal is wrote to both live and backup.
    
    Say we set both syncNonTransactional and syncTransactional to true(the default), the commit/rollback/the last addref at live will be synced.
    
    As Andy said, if we do not sync on backup side, no extra guarantees. However, if we can tell the backup to sync, does this makes sense? We need add extra property `sync` on ReplicationXXXMessage.
    
    Of cause it kills perf. But Artemis is distributed message system, we can add more nodes to cluster.
    
    Any idea?


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by mtaylor <gi...@git.apache.org>.

Github user mtaylor commented on the issue:

https://github.com/apache/activemq-artemis/pull/1747

I think it's a good idea to understand what we failures we can recover from and which ones we can not in the context of consistency and availability. Note, I'm not including network issues here.

Availability: This PR doesn't really concern availability but for context: A single live/backup pair setup can recover from a *single* machine failure. In the event of a live failure, the backup takes over, (it may have some data in memory at this point). In the event of a backup failure, availability is preserved.

Consistency: Consistency is preserved even if both machines fail. The ack isn't sent back to the client unless it's persisted on the live. Should live fail, the backup is still consistent, (though there is a small window where data is in memory). Should the backup fail before it syncs to disk, it will not have consistency, however, the live still has the copy of it's journal which is up to date.

There is a known issue here in that the broker with the most recent journal must be started first after a live and backup failure (this goes beyond this case) and required manual intervention to deal with. We will be addressing this soon.

There is however still a use case for this patch. In that it does allow recovery from a disk failure should both live and backup fail. The probability is likely very small, but I can see a case for ensuring data is persisted in two places, but not at the expense of performance for other use cases.

This could be a configuration option. However, for this use case I would recommend using a shared store solution as dedicated shared storage solutions can already address these issues.

---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by michaelandrepearce <gi...@git.apache.org>.

Github user michaelandrepearce commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    I agree with Andy, this would add latency overhead, killing perf.
    
    What is the issue you’re trying to solve?


---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by michaelandrepearce <gi...@git.apache.org>.

Github user michaelandrepearce commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    @yourcaptain as noted though if you read and understand the steps Andy (and myself) have detailed this gives no further guarantees of data loss.
    
    Please explain (even better provide an integration test) the scenario you see where data loss would occur?
    



---

[GitHub] activemq-artemis issue #1747: [WIP] NO-JIRA DISCUSS Replica use operationcon...

Posted by michaelandrepearce <gi...@git.apache.org>.

Github user michaelandrepearce commented on the issue:

    https://github.com/apache/activemq-artemis/pull/1747
  
    I so first this doesn’t mean that the replication needs to occur after sync to disk. They can continue in parallel.
    
    You simply want the slave to fsync  before replying success of replication. Also note this should be togglable with the current behaviour being default to not effect existing users, who simply need to have protection from a single failure in HA mode. eg you have to toggle this on if you want this.
    
    Lastly I’m not sure what this brings you still, as soon as slave activates on master failure it fsyncs it’s journal to disk as part of becoming active. you should note that if you’re actual requirement is you want to protect from data loss in case of two node failure you can simply add more slave nodes. So in case Master died, and then the first slave failed you would still have the second slave with the data (and now active) and on activation would fsync to disk. 
    
    
    



---