You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org> on 2016/02/04 07:09:38 UTC
Change in asterixdb[master]: Asterix NCs Failback Support
Murtadha Hubail has uploaded a new change for review.
https://asterix-gerrit.ics.uci.edu/613
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
This change include the following:
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Prevent logs and files of tookover partitions from being replicated.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-external-data/src/main/java/org/apache/asterix/external/util/NodeResolver.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vagrant_script.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vagrant_mgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
86 files changed, 3,153 insertions(+), 2,129 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/1
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "abdullah alamoudi (Code Review)" <do...@asterixdb.incubator.apache.org>.
abdullah alamoudi has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 2: Verified+1
Build Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/630/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 6: Verified+1
Build Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/727/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 5:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/721/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 2:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/630/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Murtadha Hubail has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 4:
(12 comments)
@Abdullah,
I addressed your comments and also updated the fully descriptive test files names that you criticised :)
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
File asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java:
Line 183: PersistentLocalResourceRepository persistentLocalResourceRepository = (PersistentLocalResourceRepository) localResourceRepository;
> instead of down casting, why not change localResourceRepository type.
Done
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
File asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java:
Line 31: public enum FailbackPlanState {
> Can you add comments to make sure the state transition is clear?
Done. I also added comments on the possible states in AsterixClusterProperties at each stage of the plan lifecycle.
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
File asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java:
Line 223: System.out.println("Repinse " + errorBody);
> Fix as you see fit.
Done
Line 265: HttpMethodBase method = new GetMethod(url);;
> remove additional semi-colon
Done
Line 527: if (queryCount == expectedResultFileCtxs.size() - 1) {
> https://asterix-gerrit.ics.uci.edu/#/c/625/12/asterix-common/src/test/java/
As we discussed, I remove this and added it to the finally block similar to your change.
Line 629: "http://" + host + ":" + port + "/admin/cluster");
> get the url from the servlet definition so if it changes, we don't need to
Done
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
File asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java:
Line 214: state = ClusterState.ACTIVE;
> change to while not 0. get rid of recursion
Done
Line 584: } else {
> fix bug. else if failed, then revert or something
Done
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java:
Line 277: break;
> doesn't look nice. find a better way??
Done
Line 300: //if the component belong to a shit
> Fix comment
Done
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java:
Line 72: int maxRecoveryAttempts = 5;
> get from a configuration file?
I changed it to be read from AsterixReplicationProperties but currently it is just a constant, but it can be read from a config file later.
Line 251: public void startFailbackProcess() {
> get from conf?
I changed it to be read from AsterixReplicationProperties but currently it is just a constant, but it can be read from a config file later.
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/613
to look at the new patch set (#4).
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Replicate files to remote replicas for partitions of interest only.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vagrant_script.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vagrant_mgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
85 files changed, 3,259 insertions(+), 2,176 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/4
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 6:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/727/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 1: Verified-1
Build Failed
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/791/ : ABORTED
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/613
to look at the new patch set (#5).
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Replicate files to remote replicas for partitions of interest only.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M .gitignore
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
A asterix-common/src/main/java/org/apache/asterix/common/utils/ServletUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
C asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vmgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.4.vscript.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.4.vscript.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/metadata_node/metadata_node.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
90 files changed, 3,399 insertions(+), 2,215 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/5
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/613
to look at the new patch set (#2).
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
This change include the following:
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Prevent logs and files of tookover partitions from being replicated.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vagrant_script.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vagrant_mgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
84 files changed, 3,139 insertions(+), 2,120 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/2
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Murtadha Hubail has submitted this change and it was merged.
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Replicate files to remote replicas for partitions of interest only.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Reviewed-on: https://asterix-gerrit.ics.uci.edu/613
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <ba...@gmail.com>
---
M .gitignore
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
A asterix-common/src/main/java/org/apache/asterix/common/utils/ServletUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
C asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vmgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.4.vscript.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.4.vscript.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/metadata_node/metadata_node.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
90 files changed, 3,401 insertions(+), 2,229 deletions(-)
Approvals:
abdullah alamoudi: Looks good to me, approved
Jenkins: Verified
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 3:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/706/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 4:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/711/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Murtadha Hubail has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 5:
(9 comments)
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
File asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java:
Line 38: * JobId(4)
> Update comment
Done
Line 67: * 3) UPDATE: 63 + PKValueSize + NewValueSize (5 + 12 + PKValueSize + 20 + 9 + NewValueSize + 8)
> Let's make this number right
Done
Line 108: private boolean replicate = false;
> s/replicate/replicated/g
Done
Line 444: writeLogRecordCommonFields(buffer);
> Should either remove this whole if block unless there's a way that handleGe
Removed and added a note to avoid future invalid uses.
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
File asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java:
Line 511: while (pendingProcessingFailbackPlans.size() > 0) {
> Use Queue methods here rather than List ones (since it's kind of a queue, e
Done
Line 663:
> Redundant, above map is complete representation of all plans
Removed.
Line 669: public boolean isMetadataNodeActive() {
> Should be synchronized
Done
Line 684: public JSONObject getClusterStateDescription() throws JSONException {
> Should also be synchronized
Done
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java:
Line 328: }
> See if this can be something less broad than Exception
Done
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Ian Maxon (Code Review)" <do...@asterixdb.incubator.apache.org>.
Ian Maxon has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 5: Code-Review+1
(9 comments)
Minor comments, +1.95
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
File asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java:
Line 38: * JobId(4)
Update comment
Line 67: * 3) UPDATE: 63 + PKValueSize + NewValueSize (5 + 12 + PKValueSize + 20 + 9 + NewValueSize + 8)
Let's make this number right
Line 108: private boolean replicate = false;
s/replicate/replicated/g
Line 444: writeLogRecordCommonFields(buffer);
Should either remove this whole if block unless there's a way that handleGetRemoteLogs() will give a record of LogType.FLUSH.
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
File asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java:
Line 511: while (pendingProcessingFailbackPlans.size() > 0) {
Use Queue methods here rather than List ones (since it's kind of a queue, even if it is a LinkedList).
Line 663:
Redundant, above map is complete representation of all plans
Line 669: public boolean isMetadataNodeActive() {
Should be synchronized
Line 684: public JSONObject getClusterStateDescription() throws JSONException {
Should also be synchronized
https://asterix-gerrit.ics.uci.edu/#/c/613/5/asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java:
Line 328: }
See if this can be something less broad than Exception
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Ian Maxon, Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/613
to look at the new patch set (#6).
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Replicate files to remote replicas for partitions of interest only.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M .gitignore
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
A asterix-common/src/main/java/org/apache/asterix/common/utils/ServletUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
C asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cstate.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vmgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.4.vscript.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.4.vscript.aql
R asterix-installer/src/test/resources/integrationts/replication/queries/failover/metadata_node/metadata_node.3.vscript.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
90 files changed, 3,401 insertions(+), 2,229 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/6
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Murtadha Hubail (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/613
to look at the new patch set (#3).
Change subject: Asterix NCs Failback Support
......................................................................
Asterix NCs Failback Support
- Allow Failed NCs to failback and takeover their partitions.
- New cluster API servlet for cluster state description.
- Remove nodeId from txn logs except remote FLUSH_LOG.
- Add partition id in UPDATE and Entity_COMMIT logs.
- Adapt remote recovery to new logs format.
- Refactor RecoveryManager and split Analysis and Redo phase.
- Spill remote recover logs to temporary file.
- Replicate files to remote replicas for partitions of interest only.
- Introduce NC active/inactive partitions concept.
- Test case for failback.
Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
---
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java
M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java
M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java
M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java
M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java
M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java
M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java
A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java
A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java
M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vagrant_script.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cluster_state.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vagrant_mgx.aql
A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql
M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm
A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm
M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml
M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java
M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java
M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java
M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
85 files changed, 3,259 insertions(+), 2,176 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/613/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 5: Verified+1
Build Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/721/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 4: Verified+1
Build Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/711/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 1:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/791/
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "abdullah alamoudi (Code Review)" <do...@asterixdb.incubator.apache.org>.
abdullah alamoudi has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 4:
(12 comments)
a few comments
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java
File asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java:
Line 183: PersistentLocalResourceRepository persistentLocalResourceRepository = (PersistentLocalResourceRepository) localResourceRepository;
instead of down casting, why not change localResourceRepository type.
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java
File asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java:
Line 31: public enum FailbackPlanState {
Can you add comments to make sure the state transition is clear?
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
File asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java:
Line 223: System.out.println("Repinse " + errorBody);
Fix as you see fit.
Line 265: HttpMethodBase method = new GetMethod(url);;
remove additional semi-colon
Line 527: if (queryCount == expectedResultFileCtxs.size() - 1) {
https://asterix-gerrit.ics.uci.edu/#/c/625/12/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
Line 629: "http://" + host + ":" + port + "/admin/cluster");
get the url from the servlet definition so if it changes, we don't need to update it here?
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java
File asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java:
Line 214: state = ClusterState.ACTIVE;
change to while not 0. get rid of recursion
Line 584: } else {
fix bug. else if failed, then revert or something
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java:
Line 277: break;
doesn't look nice. find a better way??
Line 300: //if the component belong to a shit
Fix comment
https://asterix-gerrit.ics.uci.edu/#/c/613/4/asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java
File asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java:
Line 72: int maxRecoveryAttempts = 5;
get from a configuration file?
Line 251: public void startFailbackProcess() {
get from conf?
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Asterix NCs Failback Support
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Asterix NCs Failback Support
......................................................................
Patch Set 3: Verified-1
Build Failed
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-topic/706/ : FAILURE
--
To view, visit https://asterix-gerrit.ics.uci.edu/613
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-Reviewer: abdullah alamoudi <ba...@gmail.com>
Gerrit-HasComments: No