[GitHub] [accumulo] dlmarion commented on a diff in pull request #3180: Enable users to provide per-volume Hadoop Filesystem overrides

["dlmarion (via GitHub)" <gi...@apache.org>]

dlmarion commented on code in PR #3180:
URL: https://github.com/apache/accumulo/pull/3180#discussion_r1097414905


##########
server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java:
##########
@@ -353,6 +362,38 @@ public short getDefaultReplication(Path path) {
     return getFileSystemByPath(path).getDefaultReplication(path);
   }
 
+  /**
+   * The Hadoop Configuration object does not currently allow for duplicate properties to be set in
+   * a single Configuration for different FileSystem URIs. Here we will look for properties in the
+   * Accumulo configuration of the form:
+   *
+   * <pre>
+   * general.custom.&lt;volume-uri&gt;.&lt;hdfs-property&gt;
+   * </pre>
+   *
+   * We will use these properties to return a new Configuration object that can be used with the
+   * FileSystem URI.
+   *
+   * @param conf AccumuloConfiguration object
+   * @param hadoopConf Hadoop Configuration object
+   * @param filesystemURI Volume Filesystem URI
+   * @return Hadoop Configuration with custom overrides for this FileSystem
+   */
+  private static Configuration getVolumeManagerConfiguration(AccumuloConfiguration conf,
+      final Configuration hadoopConf, final String filesystemURI) {
+    final Configuration volumeConfig = new Configuration(hadoopConf);
+    final Map<String,String> customProps =
+        conf.getAllPropertiesWithPrefixStripped(Property.GENERAL_ARBITRARY_PROP_PREFIX);
+    customProps.forEach((key, value) -> {
+      if (key.startsWith(filesystemURI)) {
+        String property = key.substring(filesystemURI.length() + 1);
+        log.debug("Overriding property {} to {} for volume {}", property, value, filesystemURI);

Review Comment:
   It should only happen once, at server startup. Increased logging level to INFO in e0c12c4.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org