You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by mh...@apache.org on 2019/06/27 16:59:17 UTC
[asterixdb] branch master updated (303d9cb -> 60ab58d)
This is an automated email from the ASF dual-hosted git repository.
mhubail pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git.
from 303d9cb [NO ISSUE] Reorganize OVER clause documentation
new 1f97881 [NO ISSUE][OTH] Use Absolute Path in FileReference hashCode/equals
new fa6ce67 [NO ISSUE][REPL] Wait For Dataset IO Before Replica Sync
new 60ab58d Merge commit 'fa6ce67' from stabilization-f69489
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/asterix/common/api/IDatasetLifecycleManager.java | 8 ++++++++
.../apache/asterix/common/context/DatasetLifecycleManager.java | 9 +++++++++
.../org/apache/asterix/replication/sync/ReplicaSynchronizer.java | 8 ++++++++
.../src/main/java/org/apache/hyracks/api/io/FileReference.java | 4 ++--
4 files changed, 27 insertions(+), 2 deletions(-)
[asterixdb] 03/03: Merge commit 'fa6ce67' from stabilization-f69489
Posted by mh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mhubail pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 60ab58df1515832f6d0b741dae05d827cc54bd4a
Merge: 303d9cb fa6ce67
Author: Murtadha Hubail <mh...@apache.org>
AuthorDate: Wed Jun 26 23:13:08 2019 +0300
Merge commit 'fa6ce67' from stabilization-f69489
Change-Id: I01e1810f1c44d9ecaf569dd22577c33e8702c682
.../org/apache/asterix/common/api/IDatasetLifecycleManager.java | 8 ++++++++
.../apache/asterix/common/context/DatasetLifecycleManager.java | 9 +++++++++
.../org/apache/asterix/replication/sync/ReplicaSynchronizer.java | 8 ++++++++
.../src/main/java/org/apache/hyracks/api/io/FileReference.java | 4 ++--
4 files changed, 27 insertions(+), 2 deletions(-)
[asterixdb] 02/03: [NO ISSUE][REPL] Wait For Dataset IO Before
Replica Sync
Posted by mh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mhubail pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit fa6ce67208d988821af03cbc94d56ce6af584873
Author: Murtadha Hubail <mh...@apache.org>
AuthorDate: Sun Jun 16 04:53:33 2019 +0300
[NO ISSUE][REPL] Wait For Dataset IO Before Replica Sync
- user model changes: no
- storage format changes: no
- interface changes: yes
Details:
- Add API in DatasetLifecycleManager to wait for IO on datasets
matching a replication strategy.
- Before synchronizing the files on a replica, wait for any
on-going IO operations on replicated datasets to make sure
the set of files won't change while the replica is being
synchronized (e.g. a merge operation won't delete a file
that we wanted to send to a replica).
Change-Id: I01ed5c9379cf7ae249faeef624d5226ea699cf22
Reviewed-on: https://asterix-gerrit.ics.uci.edu/3446
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <mb...@apache.org>
---
.../org/apache/asterix/common/api/IDatasetLifecycleManager.java | 8 ++++++++
.../apache/asterix/common/context/DatasetLifecycleManager.java | 9 +++++++++
.../org/apache/asterix/replication/sync/ReplicaSynchronizer.java | 8 ++++++++
3 files changed, 25 insertions(+)
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
index d18b6ab..0cc3d6d 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java
@@ -133,4 +133,12 @@ public interface IDatasetLifecycleManager extends IResourceLifecycleManager<IInd
* @throws HyracksDataException
*/
void flushDataset(IReplicationStrategy replicationStrategy) throws HyracksDataException;
+
+ /**
+ * Waits for all ongoing IO operations on all open datasets that are matching {@code replicationStrategy}.
+ *
+ * @param replicationStrategy
+ * @throws HyracksDataException
+ */
+ void waitForIO(IReplicationStrategy replicationStrategy) throws HyracksDataException;
}
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
index d767219..30b27d8 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
@@ -599,6 +599,15 @@ public class DatasetLifecycleManager implements IDatasetLifecycleManager, ILifeC
}
}
+ @Override
+ public void waitForIO(IReplicationStrategy replicationStrategy) throws HyracksDataException {
+ for (DatasetResource dsr : datasets.values()) {
+ if (dsr.isOpen() && replicationStrategy.isMatch(dsr.getDatasetID())) {
+ dsr.getDatasetInfo().waitForIO();
+ }
+ }
+ }
+
private void closeIndex(IndexInfo indexInfo) throws HyracksDataException {
if (indexInfo.isOpen()) {
ILSMOperationTracker opTracker = indexInfo.getIndex().getOperationTracker();
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
index 09f1205..0f0b5bd 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
@@ -53,10 +53,12 @@ public class ReplicaSynchronizer {
private void syncFiles() throws IOException {
final ReplicaFilesSynchronizer fileSync = new ReplicaFilesSynchronizer(appCtx, replica);
+ waitForReplicatedDatasetsIO();
fileSync.sync();
// flush replicated dataset to generate disk component for any remaining in-memory components
final IReplicationStrategy replStrategy = appCtx.getReplicationManager().getReplicationStrategy();
appCtx.getDatasetLifecycleManager().flushDataset(replStrategy);
+ waitForReplicatedDatasetsIO();
// sync any newly generated files
fileSync.sync();
}
@@ -75,4 +77,10 @@ public class ReplicaSynchronizer {
(PersistentLocalResourceRepository) appCtx.getLocalResourceRepository();
return localResourceRepository.getReplicatedIndexesMaxComponentId(partition, replStrategy);
}
+
+ private void waitForReplicatedDatasetsIO() throws HyracksDataException {
+ // wait for IO operations to ensure replicated datasets files won't change during replica sync
+ final IReplicationStrategy replStrategy = appCtx.getReplicationManager().getReplicationStrategy();
+ appCtx.getDatasetLifecycleManager().waitForIO(replStrategy);
+ }
}
[asterixdb] 01/03: [NO ISSUE][OTH] Use Absolute Path in
FileReference hashCode/equals
Posted by mh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mhubail pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 1f978819e3a86ae10d77d3211964145b68d9e05a
Author: Murtadha Hubail <mh...@apache.org>
AuthorDate: Fri Jun 7 06:38:50 2019 +0300
[NO ISSUE][OTH] Use Absolute Path in FileReference hashCode/equals
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- To avoid cases where unintentional false comparison result happens
due to a relative path of one FileReference starting with "/" and
the other's does not, use the file absolute path in hashCode and
equals.
Change-Id: Idcb642b3c5396a8319dc0aaf050e8af6fca93039
Reviewed-on: https://asterix-gerrit.ics.uci.edu/3430
Reviewed-by: Michael Blow <mb...@apache.org>
Sonar-Qube: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
.../src/main/java/org/apache/hyracks/api/io/FileReference.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
index 4ded855..e4097ff 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
@@ -58,12 +58,12 @@ public final class FileReference implements Serializable {
if (!(o instanceof FileReference)) {
return false;
}
- return path.equals(((FileReference) o).path) && dev.equals(((FileReference) o).dev);
+ return file.getAbsolutePath().equals(((FileReference) o).getAbsolutePath());
}
@Override
public int hashCode() {
- return path.hashCode();
+ return file.getAbsolutePath().hashCode();
}
/**