You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by mb...@apache.org on 2020/02/09 21:38:14 UTC
[asterixdb] branch master updated (e74454b -> 4a6cb60)
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git.
from e74454b [NO ISSUE][MTD] Remove '$' from function parameter names
new 696fc9a [NO ISSUE][*DB][CLUS] Ensure registering node is authorized
new d37b7f7 Merge branch 'gerrit/mad-hatter'
new aa02802 [NO ISSUE][HYR] Fix build warning
new 82319bc [NO ISSUE][*DB][API] Return 503 on executor shutdown
new 8d1e00f [NO ISSUE][NET] Re-resolve InetSocketAddresses on failure
new b2ae954 Merge branch 'mad-hatter'
new 4a6cb60 Merge branch 'gerrit/march-hare'
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../api/http/server/DiagnosticsApiServlet.java | 10 +++++--
.../app/replication/NcLifecycleCoordinator.java | 22 +++++++++++----
.../asterix/runtime/ClusterStateManagerTest.java | 4 +++
.../common/cluster/IClusterStateManager.java | 2 --
.../replication/IReplicationDestination.java | 4 +--
.../asterix/common/storage/ReplicaIdentifier.java | 10 ++++++-
.../asterix/replication/api/PartitionReplica.java | 11 +++++++-
.../replication/api/ReplicationDestination.java | 32 ++++++++++++++++------
.../replication/management/ReplicationManager.java | 5 ++--
.../replication/sync/ReplicaSynchronizer.java | 3 +-
hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml | 2 +-
.../hyracks/ipc/impl/ReconnectingIPCHandle.java | 3 +-
.../java/org/apache/hyracks/util/NetworkUtil.java | 19 +++++++++++++
13 files changed, 98 insertions(+), 29 deletions(-)
[asterixdb] 07/07: Merge branch 'gerrit/march-hare'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 4a6cb608a5791cdfb44704286a2a568d488097f1
Merge: e74454b b2ae954
Author: Michael Blow <mb...@apache.org>
AuthorDate: Sun Feb 9 14:13:01 2020 -0500
Merge branch 'gerrit/march-hare'
Change-Id: I477a59907d5d20aa88202c6fb1fa2b8af165e6b9
.../api/http/server/DiagnosticsApiServlet.java | 10 +++++--
.../app/replication/NcLifecycleCoordinator.java | 22 +++++++++++----
.../asterix/runtime/ClusterStateManagerTest.java | 4 +++
.../common/cluster/IClusterStateManager.java | 2 --
.../replication/IReplicationDestination.java | 4 +--
.../asterix/common/storage/ReplicaIdentifier.java | 10 ++++++-
.../asterix/replication/api/PartitionReplica.java | 11 +++++++-
.../replication/api/ReplicationDestination.java | 32 ++++++++++++++++------
.../replication/management/ReplicationManager.java | 5 ++--
.../replication/sync/ReplicaSynchronizer.java | 3 +-
hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml | 2 +-
.../hyracks/ipc/impl/ReconnectingIPCHandle.java | 3 +-
.../java/org/apache/hyracks/util/NetworkUtil.java | 19 +++++++++++++
13 files changed, 98 insertions(+), 29 deletions(-)
diff --cc hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
index 228fd91,3f6e90c..1fcd806
--- a/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
+++ b/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
@@@ -27,8 -27,8 +27,9 @@@ import java.net.URISyntaxException
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
+ import java.util.Objects;
import javax.net.ssl.SSLEngine;
[asterixdb] 03/07: [NO ISSUE][HYR] Fix build warning
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit aa028027a2a0566b17141ebbb9f8500c565697d5
Author: Michael Blow <mb...@apache.org>
AuthorDate: Tue Feb 4 21:45:02 2020 -0500
[NO ISSUE][HYR] Fix build warning
[WARNING] The artifact xml-apis:xml-apis:jar:2.0.2 has been relocated to xml-apis:xml-apis:jar:1.0.b2
[WARNING] While downloading xml-apis:xml-apis:2.0.2
Change-Id: Ibda23a2e8308937f343d80eff04ede9a235da0d5
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4904
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <mb...@apache.org>
Reviewed-by: Hussain Towaileb <hu...@gmail.com>
Reviewed-by: Ian Maxon <im...@uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml b/hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml
index 7e00d82..9528eaa 100644
--- a/hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml
@@ -205,7 +205,7 @@
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
- <version>2.0.2</version>
+ <version>1.0.b2</version>
<scope>test</scope>
</dependency>
</dependencies>
[asterixdb] 04/07: [NO ISSUE][*DB][API] Return 503 on executor
shutdown
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 82319bc03d9da3edd11d7b4f8d2d4b6c50e4e1a4
Author: Michael Blow <mb...@apache.org>
AuthorDate: Wed Feb 5 10:23:31 2020 -0500
[NO ISSUE][*DB][API] Return 503 on executor shutdown
Change-Id: I8efd7dc9a8f19972884e542dd97906d4701661b8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4944
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <ti...@apache.org>
---
.../apache/asterix/api/http/server/DiagnosticsApiServlet.java | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/DiagnosticsApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/DiagnosticsApiServlet.java
index e294510..9876eed 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/DiagnosticsApiServlet.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/DiagnosticsApiServlet.java
@@ -32,6 +32,7 @@ import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
import org.apache.asterix.common.cluster.IClusterStateManager;
import org.apache.asterix.common.dataflow.ICcApplicationContext;
@@ -75,10 +76,13 @@ public class DiagnosticsApiServlet extends NodeControllerDetailsApiServlet {
response.setStatus(HttpResponseStatus.SERVICE_UNAVAILABLE);
} catch (IllegalArgumentException e) { // NOSONAR - exception not logged or rethrown
response.setStatus(HttpResponseStatus.NOT_FOUND);
+ } catch (RejectedExecutionException e) {
+ // we must be shutting down, return 503
+ LOGGER.info("RejectedExecutionException while servicing request; returning 503", e);
+ sendError(response, HttpResponseStatus.SERVICE_UNAVAILABLE, null);
} catch (Exception e) {
- LOGGER.log(Level.INFO, "exception thrown for " + request, e);
- response.setStatus(HttpResponseStatus.INTERNAL_SERVER_ERROR);
- responseWriter.write(e.toString());
+ LOGGER.warn("exception while servicing request; returning 500", e);
+ sendError(response, HttpResponseStatus.INTERNAL_SERVER_ERROR, e.toString());
}
responseWriter.flush();
}
[asterixdb] 06/07: Merge branch 'mad-hatter'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit b2ae95475baf500c8cdfe7184ff6302b2d4038aa
Merge: d37b7f7 8d1e00f
Author: Michael Blow <mb...@apache.org>
AuthorDate: Sun Feb 9 09:07:51 2020 -0500
Merge branch 'mad-hatter'
Change-Id: Ief0a2fe1f0cc29411eb52b52506a5bc1a2a3fd99
asterixdb/NOTICE | 2 +-
.../api/http/server/DiagnosticsApiServlet.java | 10 +++++--
.../replication/IReplicationDestination.java | 4 +--
.../asterix/common/storage/ReplicaIdentifier.java | 10 ++++++-
.../asterix/replication/api/PartitionReplica.java | 11 +++++++-
.../replication/api/ReplicationDestination.java | 32 ++++++++++++++++------
.../replication/management/ReplicationManager.java | 5 ++--
.../replication/sync/ReplicaSynchronizer.java | 3 +-
hyracks-fullstack/NOTICE | 2 +-
hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml | 2 +-
.../hyracks/ipc/impl/ReconnectingIPCHandle.java | 3 +-
.../java/org/apache/hyracks/util/NetworkUtil.java | 19 +++++++++++++
12 files changed, 80 insertions(+), 23 deletions(-)
[asterixdb] 05/07: [NO ISSUE][NET] Re-resolve InetSocketAddresses
on failure
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 8d1e00f7f7307f33f44493a4b33b1e2fe0989ed5
Author: Michael Blow <mb...@apache.org>
AuthorDate: Fri Feb 7 19:31:49 2020 -0500
[NO ISSUE][NET] Re-resolve InetSocketAddresses on failure
Support IP address updates on replicas & CC by re-resolving the IP address
on connection failure
Change-Id: I2532a27c2abb8b891df668b0adc95cc33da99620
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4965
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <mb...@apache.org>
Reviewed-by: Murtadha Hubail <mh...@apache.org>
---
asterixdb/NOTICE | 2 +-
.../replication/IReplicationDestination.java | 4 +--
.../asterix/common/storage/ReplicaIdentifier.java | 10 ++++++-
.../asterix/replication/api/PartitionReplica.java | 11 +++++++-
.../replication/api/ReplicationDestination.java | 32 ++++++++++++++++------
.../replication/management/ReplicationManager.java | 5 ++--
.../replication/sync/ReplicaSynchronizer.java | 3 +-
hyracks-fullstack/NOTICE | 2 +-
.../hyracks/ipc/impl/ReconnectingIPCHandle.java | 3 +-
.../java/org/apache/hyracks/util/NetworkUtil.java | 19 +++++++++++++
10 files changed, 72 insertions(+), 19 deletions(-)
diff --git a/asterixdb/NOTICE b/asterixdb/NOTICE
index 7615782..b4729a8 100644
--- a/asterixdb/NOTICE
+++ b/asterixdb/NOTICE
@@ -1,5 +1,5 @@
Apache AsterixDB
-Copyright 2015-2019 The Apache Software Foundation
+Copyright 2015-2020 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationDestination.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationDestination.java
index 2fe9de8..f835c43 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationDestination.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationDestination.java
@@ -52,9 +52,9 @@ public interface IReplicationDestination {
Set<IPartitionReplica> getReplicas();
/**
- * Gets the location of this {@link IReplicationDestination}
+ * Gets the (resolved) location of this {@link IReplicationDestination}
*
- * @return the location
+ * @return the (resolved) location
*/
InetSocketAddress getLocation();
}
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ReplicaIdentifier.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ReplicaIdentifier.java
index f68ad09..c4bb74c 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ReplicaIdentifier.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ReplicaIdentifier.java
@@ -20,11 +20,13 @@ package org.apache.asterix.common.storage;
import java.net.InetSocketAddress;
+import org.apache.hyracks.util.NetworkUtil;
+
public class ReplicaIdentifier {
private final int partition;
- private final InetSocketAddress location;
private final String id;
+ private volatile InetSocketAddress location;
private ReplicaIdentifier(int partition, InetSocketAddress location) {
this.partition = partition;
@@ -44,6 +46,12 @@ public class ReplicaIdentifier {
return location;
}
+ public InetSocketAddress refreshLocation() {
+ //noinspection NonAtomicOperationOnVolatileField
+ location = NetworkUtil.refresh(location);
+ return location;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/PartitionReplica.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/PartitionReplica.java
index e81c25a..f2d2496 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/PartitionReplica.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/PartitionReplica.java
@@ -97,7 +97,7 @@ public class PartitionReplica implements IPartitionReplica {
public synchronized ISocketChannel getChannel() {
try {
if (!NetworkingUtil.isHealthy(sc)) {
- sc = ReplicationProtocol.establishReplicaConnection(appCtx, id.getLocation());
+ establishReplicaConnection();
}
return sc;
} catch (IOException e) {
@@ -105,6 +105,15 @@ public class PartitionReplica implements IPartitionReplica {
}
}
+ private void establishReplicaConnection() throws IOException {
+ try {
+ sc = ReplicationProtocol.establishReplicaConnection(appCtx, id.getLocation());
+ } catch (Exception e) {
+ // try to re-resolve the address, in case our replica has had his IP address updated
+ sc = ReplicationProtocol.establishReplicaConnection(appCtx, id.refreshLocation());
+ }
+ }
+
public synchronized void close() {
try {
if (sc != null) {
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/ReplicationDestination.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/ReplicationDestination.java
index ef1bc28..782a801 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/ReplicationDestination.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/api/ReplicationDestination.java
@@ -32,6 +32,7 @@ import org.apache.asterix.common.replication.IReplicationDestination;
import org.apache.asterix.replication.management.NetworkingUtil;
import org.apache.asterix.replication.messaging.ReplicationProtocol;
import org.apache.hyracks.api.network.ISocketChannel;
+import org.apache.hyracks.util.NetworkUtil;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -39,15 +40,20 @@ public class ReplicationDestination implements IReplicationDestination {
private static final Logger LOGGER = LogManager.getLogger();
private final Set<IPartitionReplica> replicas = new HashSet<>();
- private final InetSocketAddress location;
+ private final InetSocketAddress inputLocation;
+ private InetSocketAddress resolvedLocation;
private ISocketChannel logRepChannel;
private ReplicationDestination(InetSocketAddress location) {
- this.location = location;
+ this.inputLocation = location;
+ this.resolvedLocation = NetworkUtil.ensureResolved(location);
}
public static ReplicationDestination at(InetSocketAddress location) {
- return new ReplicationDestination(location);
+ if (!location.isUnresolved()) {
+ throw new IllegalArgumentException("only unresolved addresses are allowed!");
+ }
+ return new ReplicationDestination(new InetSocketAddress(location.getHostString(), location.getPort()));
}
@Override
@@ -79,7 +85,7 @@ public class ReplicationDestination implements IReplicationDestination {
public synchronized ISocketChannel getLogReplicationChannel(INcApplicationContext appCtx) {
try {
if (!NetworkingUtil.isHealthy(logRepChannel)) {
- logRepChannel = ReplicationProtocol.establishReplicaConnection(appCtx, location);
+ establishReplicaConnection(appCtx);
}
return logRepChannel;
} catch (IOException e) {
@@ -87,6 +93,16 @@ public class ReplicationDestination implements IReplicationDestination {
}
}
+ protected void establishReplicaConnection(INcApplicationContext appCtx) throws IOException {
+ try {
+ logRepChannel = ReplicationProtocol.establishReplicaConnection(appCtx, resolvedLocation);
+ } catch (Exception e) {
+ // try to re-resolve the address, in case our replica has had his IP address updated
+ resolvedLocation = NetworkUtil.refresh(resolvedLocation);
+ logRepChannel = ReplicationProtocol.establishReplicaConnection(appCtx, resolvedLocation);
+ }
+ }
+
private synchronized void closeLogReplicationChannel() {
try {
if (logRepChannel != null && logRepChannel.getSocketChannel().isOpen()) {
@@ -101,7 +117,7 @@ public class ReplicationDestination implements IReplicationDestination {
@Override
public InetSocketAddress getLocation() {
- return location;
+ return resolvedLocation;
}
@Override
@@ -113,16 +129,16 @@ public class ReplicationDestination implements IReplicationDestination {
return false;
}
ReplicationDestination that = (ReplicationDestination) o;
- return Objects.equals(location, that.location);
+ return Objects.equals(inputLocation, that.inputLocation);
}
@Override
public String toString() {
- return location.toString();
+ return resolvedLocation.toString();
}
@Override
public int hashCode() {
- return Objects.hash(location);
+ return Objects.hash(inputLocation);
}
}
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
index bd99ec4..7ed674e 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
@@ -34,6 +34,7 @@ import org.apache.asterix.common.replication.ReplicationStrategyFactory;
import org.apache.asterix.common.transactions.ILogRecord;
import org.apache.asterix.replication.api.ReplicationDestination;
import org.apache.hyracks.api.replication.IReplicationJob;
+import org.apache.hyracks.util.NetworkUtil;
import org.apache.hyracks.util.annotations.ThreadSafe;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -60,7 +61,7 @@ public class ReplicationManager implements IReplicationManager {
@Override
public void register(IPartitionReplica replica) {
synchronized (dests) {
- final InetSocketAddress location = replica.getIdentifier().getLocation();
+ final InetSocketAddress location = NetworkUtil.ensureUnresolved(replica.getIdentifier().getLocation());
final ReplicationDestination replicationDest = dests.computeIfAbsent(location, ReplicationDestination::at);
replicationDest.add(replica);
logReplicationManager.register(replicationDest);
@@ -71,7 +72,7 @@ public class ReplicationManager implements IReplicationManager {
@Override
public void unregister(IPartitionReplica replica) {
synchronized (dests) {
- final InetSocketAddress location = replica.getIdentifier().getLocation();
+ final InetSocketAddress location = NetworkUtil.ensureUnresolved(replica.getIdentifier().getLocation());
final ReplicationDestination dest = dests.get(location);
if (dest == null) {
LOGGER.warn(() -> "Asked to unregister unknown replica " + replica);
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
index 123709b..261236c 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaSynchronizer.java
@@ -44,8 +44,7 @@ public class ReplicaSynchronizer {
}
public void sync() throws IOException {
- final Object syncLock = appCtx.getReplicaManager().getReplicaSyncLock();
- synchronized (syncLock) {
+ synchronized (appCtx.getReplicaManager().getReplicaSyncLock()) {
final ICheckpointManager checkpointManager = appCtx.getTransactionSubsystem().getCheckpointManager();
try {
// suspend checkpointing datasets to prevent async IO operations while sync'ing replicas
diff --git a/hyracks-fullstack/NOTICE b/hyracks-fullstack/NOTICE
index 77f31ad..95fe98a 100644
--- a/hyracks-fullstack/NOTICE
+++ b/hyracks-fullstack/NOTICE
@@ -1,5 +1,5 @@
Apache Hyracks and Algebricks
-Copyright 2015-2019 The Apache Software Foundation
+Copyright 2015-2020 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
diff --git a/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/impl/ReconnectingIPCHandle.java b/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/impl/ReconnectingIPCHandle.java
index db0ed6b..a3578ad 100644
--- a/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/impl/ReconnectingIPCHandle.java
+++ b/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/impl/ReconnectingIPCHandle.java
@@ -23,6 +23,7 @@ import java.net.InetSocketAddress;
import org.apache.hyracks.ipc.api.IIPCEventListener;
import org.apache.hyracks.ipc.api.IIPCHandle;
import org.apache.hyracks.ipc.exceptions.IPCException;
+import org.apache.hyracks.util.NetworkUtil;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -78,7 +79,7 @@ class ReconnectingIPCHandle implements IIPCHandle {
}
LOGGER.warn("ipcHandle {} disconnected; will attempt to reconnect {} times", delegate, reconnectAttempts);
listener.ipcHandleDisconnected(delegate);
- delegate = ipc.getHandle(getRemoteAddress(), reconnectAttempts);
+ delegate = ipc.getHandle(NetworkUtil.refresh(getRemoteAddress()), reconnectAttempts);
LOGGER.warn("ipcHandle {} restored", delegate);
listener.ipcHandleRestored(delegate);
return delegate;
diff --git a/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java b/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
index 763319f..3f6e90c 100644
--- a/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
+++ b/hyracks-fullstack/hyracks/hyracks-util/src/main/java/org/apache/hyracks/util/NetworkUtil.java
@@ -28,6 +28,7 @@ import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
import java.util.ArrayList;
import java.util.List;
+import java.util.Objects;
import javax.net.ssl.SSLEngine;
@@ -130,4 +131,22 @@ public class NetworkUtil {
enlargedBuffer.put(src);
return enlargedBuffer;
}
+
+ public static InetSocketAddress ensureUnresolved(InetSocketAddress address) {
+ return address.isUnresolved() ? address
+ : InetSocketAddress.createUnresolved(address.getHostString(), address.getPort());
+ }
+
+ public static InetSocketAddress ensureResolved(InetSocketAddress address) {
+ return address.isUnresolved() ? new InetSocketAddress(address.getHostString(), address.getPort()) : address;
+ }
+
+ public static InetSocketAddress refresh(InetSocketAddress original) {
+ InetSocketAddress refreshed = new InetSocketAddress(original.getHostString(), original.getPort());
+ if (!Objects.equals(original.getAddress(), refreshed.getAddress())) {
+ LOGGER.warn("ip address updated on refresh (was: {}, now: {})", original.getAddress(),
+ refreshed.getAddress());
+ }
+ return refreshed;
+ }
}
[asterixdb] 01/07: [NO ISSUE][*DB][CLUS] Ensure registering node is
authorized
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 696fc9ac9e97e1187fb15ab2dcbe3780846e8647
Author: Michael Blow <mb...@apache.org>
AuthorDate: Sat Feb 1 22:17:30 2020 -0500
[NO ISSUE][*DB][CLUS] Ensure registering node is authorized
Change-Id: I1c47e8e7934b87b758b32e186b613b33d852066d
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4863
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Michael Blow <mb...@apache.org>
Reviewed-by: Michael Blow <mb...@apache.org>
---
.../app/replication/NcLifecycleCoordinator.java | 22 ++++++++++++++++------
.../asterix/runtime/ClusterStateManagerTest.java | 4 ++++
.../common/cluster/IClusterStateManager.java | 2 --
3 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/replication/NcLifecycleCoordinator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/replication/NcLifecycleCoordinator.java
index 18191d6..0d86cb9 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/replication/NcLifecycleCoordinator.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/replication/NcLifecycleCoordinator.java
@@ -20,6 +20,7 @@ package org.apache.asterix.app.replication;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@@ -51,8 +52,9 @@ import org.apache.asterix.common.transactions.IRecoveryManager.SystemState;
import org.apache.asterix.metadata.MetadataManager;
import org.apache.hyracks.api.application.ICCServiceContext;
import org.apache.hyracks.api.client.NodeStatus;
+import org.apache.hyracks.api.control.IGatekeeper;
import org.apache.hyracks.api.exceptions.HyracksDataException;
-import org.apache.logging.log4j.Level;
+import org.apache.hyracks.control.cc.ClusterControllerService;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -61,13 +63,16 @@ public class NcLifecycleCoordinator implements INcLifecycleCoordinator {
private static final Logger LOGGER = LogManager.getLogger();
protected IClusterStateManager clusterManager;
protected volatile String metadataNodeId;
- protected Set<String> pendingStartupCompletionNodes = new HashSet<>();
+ protected Set<String> pendingStartupCompletionNodes = Collections.synchronizedSet(new HashSet<>());
protected final ICCMessageBroker messageBroker;
private final boolean replicationEnabled;
+ private final IGatekeeper gatekeeper;
public NcLifecycleCoordinator(ICCServiceContext serviceCtx, boolean replicationEnabled) {
this.messageBroker = (ICCMessageBroker) serviceCtx.getMessageBroker();
this.replicationEnabled = replicationEnabled;
+ this.gatekeeper =
+ ((ClusterControllerService) serviceCtx.getControllerService()).getApplication().getGatekeeper();
}
@Override
@@ -120,7 +125,14 @@ public class NcLifecycleCoordinator implements INcLifecycleCoordinator {
}
private void process(NCLifecycleTaskReportMessage msg) throws HyracksDataException {
- pendingStartupCompletionNodes.remove(msg.getNodeId());
+ if (!pendingStartupCompletionNodes.remove(msg.getNodeId())) {
+ LOGGER.warn("Received unexpected startup completion message from node {}", msg.getNodeId());
+ }
+ if (!gatekeeper.isAuthorized(msg.getNodeId())) {
+ LOGGER.warn("Node {} lost authorization before startup completed; ignoring registration result",
+ msg.getNodeId());
+ return;
+ }
if (msg.isSuccess()) {
clusterManager.updateNodeState(msg.getNodeId(), true, msg.getLocalCounters());
if (msg.getNodeId().equals(metadataNodeId)) {
@@ -128,9 +140,7 @@ public class NcLifecycleCoordinator implements INcLifecycleCoordinator {
}
clusterManager.refreshState();
} else {
- if (LOGGER.isErrorEnabled()) {
- LOGGER.log(Level.ERROR, msg.getNodeId() + " failed to complete startup. ", msg.getException());
- }
+ LOGGER.error("Node {} failed to complete startup", msg.getNodeId(), msg.getException());
}
}
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ClusterStateManagerTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ClusterStateManagerTest.java
index b7a8c63..9cc295e 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ClusterStateManagerTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ClusterStateManagerTest.java
@@ -32,6 +32,7 @@ import org.apache.asterix.common.cluster.IGlobalRecoveryManager;
import org.apache.asterix.common.config.MetadataProperties;
import org.apache.asterix.common.metadata.IMetadataBootstrap;
import org.apache.asterix.common.utils.NcLocalCounters;
+import org.apache.asterix.hyracks.bootstrap.CCApplication;
import org.apache.asterix.runtime.transaction.ResourceIdManager;
import org.apache.asterix.runtime.utils.BulkTxnIdFactory;
import org.apache.asterix.runtime.utils.CcApplicationContext;
@@ -211,10 +212,13 @@ public class ClusterStateManagerTest {
final ClusterControllerService ccs = Mockito.mock(ClusterControllerService.class);
JobIdFactory jobIdFactory = new JobIdFactory(CcId.valueOf(0));
Mockito.when(ccs.getJobIdFactory()).thenReturn(jobIdFactory);
+ final CCApplication ccApplication = Mockito.mock(CCApplication.class);
+ Mockito.when(ccs.getApplication()).thenReturn(ccApplication);
Mockito.when(iccServiceContext.getAppConfig()).thenReturn(applicationConfig);
Mockito.when(iccServiceContext.getControllerService()).thenReturn(ccs);
Mockito.when(ccApplicationContext.getServiceContext()).thenReturn(iccServiceContext);
+ Mockito.when(ccApplication.getGatekeeper()).thenReturn(nodeId -> true);
NcLifecycleCoordinator coordinator =
new NcLifecycleCoordinator(ccApplicationContext.getServiceContext(), false);
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/cluster/IClusterStateManager.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/cluster/IClusterStateManager.java
index 6c39372..a37e6e4 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/cluster/IClusterStateManager.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/cluster/IClusterStateManager.java
@@ -120,7 +120,6 @@ public interface IClusterStateManager {
/**
* Register the specified node partitions with the specified nodeId with this cluster state manager
- * then calls {@link IClusterStateManager#refreshState()}
*
* @param nodeId
* @param nodePartitions
@@ -130,7 +129,6 @@ public interface IClusterStateManager {
/**
* De-register the specified node's partitions from this cluster state manager
- * then calls {@link IClusterStateManager#refreshState()}
*
* @param nodeId
* @throws HyracksDataException
[asterixdb] 02/07: Merge branch 'gerrit/mad-hatter'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit d37b7f766f44a39639dc6fcface8b38376adef5c
Merge: c605728 696fc9a
Author: Michael Blow <mb...@apache.org>
AuthorDate: Tue Feb 4 15:59:16 2020 -0500
Merge branch 'gerrit/mad-hatter'
Change-Id: If9d5238e4c3ae94eb17b951f07259432b85ead9d
.../app/replication/NcLifecycleCoordinator.java | 22 ++++++++++++++++------
.../asterix/runtime/ClusterStateManagerTest.java | 4 ++++
.../common/cluster/IClusterStateManager.java | 2 --
3 files changed, 20 insertions(+), 8 deletions(-)