You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/08/16 19:01:34 UTC
[GitHub] [accumulo] milleruntime opened a new pull request #2233: Drop ServerUtil and static state
milleruntime opened a new pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233
* Move static state from ServerUtil in to ServerConstants object methods
* Move creation of ServerConstants object into ServerInfo
* Move upgrade code from ServerUtil into UpgradeCoordinator
* Move methods dealing with server stuff to ServerContext
* Create getter for ServerConstants in ServerContext
* Move AccumuloTest to manager to avoid adding a dependency
* Drop ServerUtilTest and move testing of method to ServerContextTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r690388163
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,137 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
Review comment:
I agree. I looked at this and I am not even sure if getChildren() will return null or empty list so I am going to just leave it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r690316919
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,139 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
+ try {
+ getZooReaderWriter().getChildren(Constants.ZROOT);
+ break;
+ } catch (InterruptedException e) {
+ // ignored
+ } catch (KeeperException ex) {
Review comment:
I am not sure. This code was just moved. This would be good to look into thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r690372398
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,137 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
Review comment:
This might be clearer if the try was lifted above the while and a local boolean used to terminate the while. The code is fine as is - I just don't use break much and it gives me a pause when I see it - more of a personal preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r690316919
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,139 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
+ try {
+ getZooReaderWriter().getChildren(Constants.ZROOT);
+ break;
+ } catch (InterruptedException e) {
+ // ignored
+ } catch (KeeperException ex) {
Review comment:
I am not sure. This code was just moved. But your suggestion is definitely better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r689816373
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,139 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
+ try {
+ getZooReaderWriter().getChildren(Constants.ZROOT);
+ break;
+ } catch (InterruptedException e) {
+ // ignored
+ } catch (KeeperException ex) {
Review comment:
Is it really appropriate to ignore the interrupt - it might be better to reset the interrupt status and then throw an exception.?
If it really is appropriate to catch and swallow the interrupt then maybe:
```suggestion
} catch (InterruptedException | KeeperException ex) {
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,139 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
+ try {
+ getZooReaderWriter().getChildren(Constants.ZROOT);
+ break;
+ } catch (InterruptedException e) {
+ // ignored
+ } catch (KeeperException ex) {
+ log.info("Waiting for accumulo to be initialized");
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ }
+ }
+ log.info("ZooKeeper connected and initialized, attempting to talk to HDFS");
+ long sleep = 1000;
+ int unknownHostTries = 3;
+ while (true) {
+ try {
+ if (getVolumeManager().isReady())
+ break;
+ log.warn("Waiting for the NameNode to leave safemode");
+ } catch (IOException ex) {
+ log.warn("Unable to connect to HDFS", ex);
+ } catch (IllegalArgumentException e) {
+ /* Unwrap the UnknownHostException so we can deal with it directly */
+ if (e.getCause() instanceof UnknownHostException) {
+ if (unknownHostTries > 0) {
+ log.warn("Unable to connect to HDFS, will retry. cause: {}", e.getCause());
+ /*
+ * We need to make sure our sleep period is long enough to avoid getting a cached
+ * failure of the host lookup.
+ */
+ int ttl = AddressUtil.getAddressCacheNegativeTtl((UnknownHostException) e.getCause());
+ sleep = Math.max(sleep, (ttl + 1) * 1000L);
+ } else {
+ log.error("Unable to connect to HDFS and exceeded the maximum number of retries.", e);
+ throw e;
+ }
+ unknownHostTries--;
+ } else {
+ throw e;
+ }
+ }
+ log.info("Backing off due to failure; current sleep period is {} seconds", sleep / 1000.);
+ sleepUninterruptibly(sleep, TimeUnit.MILLISECONDS);
+ /* Back off to give transient failures more time to clear. */
+ sleep = Math.min(60 * 1000, sleep * 2);
+ }
+ log.info("Connected to HDFS");
+ }
+
+ /**
+ * Wait for ZK and hdfs, check data version and some properties, and start thread to monitor
+ * swappiness. Should only be called once during server start up.
+ */
+ public void init(String application) {
+ final AccumuloConfiguration conf = getConfiguration();
+
+ log.info("{} starting", application);
+ log.info("Instance {}", getInstanceID());
+ // It doesn't matter which Volume is used as they should all have the data version stored
+ int dataVersion = serverConstants.getAccumuloPersistentVersion(getVolumeManager().getFirst());
+ log.info("Data Version {}", dataVersion);
+ waitForZookeeperAndHdfs();
+
+ ensureDataVersionCompatible(dataVersion);
+
+ TreeMap<String,String> sortedProps = new TreeMap<>();
+ for (Map.Entry<String,String> entry : conf)
+ sortedProps.put(entry.getKey(), entry.getValue());
+
+ for (Map.Entry<String,String> entry : sortedProps.entrySet()) {
Review comment:
Is the sort for readability so that the messages reported are consistently ordered? Otherwise, it looks like the check could be done on iterating through the conf entries themselves.
Also, if the sort is desired, would
```
TreeMap<String,String> sortedProps = new TreeMap(conf);
```
work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233#discussion_r690315643
##########
File path: server/base/src/main/java/org/apache/accumulo/server/ServerContext.java
##########
@@ -279,4 +300,139 @@ public Ample getAmple() {
public Set<String> getRecoveryDirs() {
return serverConstants.getRecoveryDirs();
}
+
+ /**
+ * Check to see if this version of Accumulo can run against or upgrade the passed in data version.
+ */
+ public static void ensureDataVersionCompatible(int dataVersion) {
+ if (!(ServerConstants.CAN_RUN.contains(dataVersion))) {
+ throw new IllegalStateException("This version of accumulo (" + Constants.VERSION
+ + ") is not compatible with files stored using data version " + dataVersion);
+ }
+ }
+
+ public void waitForZookeeperAndHdfs() {
+ log.info("Attempting to talk to zookeeper");
+ while (true) {
+ try {
+ getZooReaderWriter().getChildren(Constants.ZROOT);
+ break;
+ } catch (InterruptedException e) {
+ // ignored
+ } catch (KeeperException ex) {
+ log.info("Waiting for accumulo to be initialized");
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ }
+ }
+ log.info("ZooKeeper connected and initialized, attempting to talk to HDFS");
+ long sleep = 1000;
+ int unknownHostTries = 3;
+ while (true) {
+ try {
+ if (getVolumeManager().isReady())
+ break;
+ log.warn("Waiting for the NameNode to leave safemode");
+ } catch (IOException ex) {
+ log.warn("Unable to connect to HDFS", ex);
+ } catch (IllegalArgumentException e) {
+ /* Unwrap the UnknownHostException so we can deal with it directly */
+ if (e.getCause() instanceof UnknownHostException) {
+ if (unknownHostTries > 0) {
+ log.warn("Unable to connect to HDFS, will retry. cause: {}", e.getCause());
+ /*
+ * We need to make sure our sleep period is long enough to avoid getting a cached
+ * failure of the host lookup.
+ */
+ int ttl = AddressUtil.getAddressCacheNegativeTtl((UnknownHostException) e.getCause());
+ sleep = Math.max(sleep, (ttl + 1) * 1000L);
+ } else {
+ log.error("Unable to connect to HDFS and exceeded the maximum number of retries.", e);
+ throw e;
+ }
+ unknownHostTries--;
+ } else {
+ throw e;
+ }
+ }
+ log.info("Backing off due to failure; current sleep period is {} seconds", sleep / 1000.);
+ sleepUninterruptibly(sleep, TimeUnit.MILLISECONDS);
+ /* Back off to give transient failures more time to clear. */
+ sleep = Math.min(60 * 1000, sleep * 2);
+ }
+ log.info("Connected to HDFS");
+ }
+
+ /**
+ * Wait for ZK and hdfs, check data version and some properties, and start thread to monitor
+ * swappiness. Should only be called once during server start up.
+ */
+ public void init(String application) {
+ final AccumuloConfiguration conf = getConfiguration();
+
+ log.info("{} starting", application);
+ log.info("Instance {}", getInstanceID());
+ // It doesn't matter which Volume is used as they should all have the data version stored
+ int dataVersion = serverConstants.getAccumuloPersistentVersion(getVolumeManager().getFirst());
+ log.info("Data Version {}", dataVersion);
+ waitForZookeeperAndHdfs();
+
+ ensureDataVersionCompatible(dataVersion);
+
+ TreeMap<String,String> sortedProps = new TreeMap<>();
+ for (Map.Entry<String,String> entry : conf)
+ sortedProps.put(entry.getKey(), entry.getValue());
+
+ for (Map.Entry<String,String> entry : sortedProps.entrySet()) {
Review comment:
> Is the sort for readability so that the messages reported are consistently ordered?
I think so. That `sortedProps` is only used in that for loop to log information about the properties.
> Also, if the sort is desired, would
TreeMap<String,String> sortedProps = new TreeMap(conf);
work?
No, it doesn't. TreeMap constructor can take a map so we would have to get the properties map, which we can only do using `getAllPropertiesWithPrefix()`. It is odd we don't have a method to get all properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime merged pull request #2233: Drop ServerUtil and static state
Posted by GitBox <gi...@apache.org>.
milleruntime merged pull request #2233:
URL: https://github.com/apache/accumulo/pull/2233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org