[GitHub] [hudi] danny0405 commented on a change in pull request #4699: [HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

[GitBox <gi...@apache.org>]

danny0405 commented on a change in pull request #4699:
URL: https://github.com/apache/hudi/pull/4699#discussion_r803550671



##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -101,7 +102,7 @@ public static TypedProperties getProps(FlinkStreamerConfig cfg) {
       return new TypedProperties();
     }
     return readConfig(
-        getHadoopConf(),
+            getHadoopConf(cfg),
         new Path(cfg.propsFilePath), cfg.configs).getProps();

Review comment:
       Fix the indentation.

##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -140,9 +141,21 @@ public static DFSPropertiesConfiguration readConfig(org.apache.hadoop.conf.Confi
     return conf;
   }
 
-  // Keep the redundant to avoid too many modifications.
   public static org.apache.hadoop.conf.Configuration getHadoopConf() {
-    return FlinkClientUtil.getHadoopConf();
+    return getHadoopConf(null);
+  }
+
+  // Keep the redundant to avoid too many modifications.
+  public static org.apache.hadoop.conf.Configuration getHadoopConf(Configuration configuration) {
+    if (configuration == null) {
+      return FlinkClientUtil.getHadoopConf();

Review comment:
       `configuration` => `conf`,
   Can we make the param `configuration` never null ?

##########
File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -660,25 +661,18 @@ private FlinkOptions() {
   // Prefix for Hoodie specific properties.
   private static final String PROPERTIES_PREFIX = "properties.";
 
-  /**
-   * Collects the config options that start with 'properties.' into a 'key'='value' list.
-   */
-  public static Map<String, String> getHoodieProperties(Map<String, String> options) {
-    return getHoodiePropertiesWithPrefix(options, PROPERTIES_PREFIX);
-  }
-
   /**
    * Collects the config options that start with specified prefix {@code prefix} into a 'key'='value' list.
    */
-  public static Map<String, String> getHoodiePropertiesWithPrefix(Map<String, String> options, String prefix) {
+  public static Map<String, String> getPropertiesWithPrefix(Map<String, String> options, String subprefix) {
     final Map<String, String> hoodieProperties = new HashMap<>();

Review comment:
       Can we use the `prefix` directly ? There is no need to prefix the option with `properties.`, prefix the option with `hadoop.` directly is okey.
   
   And can we also add a too method named `getHadoopOptions(Configuration conf)` here ?

##########
File path: hudi-flink/src/test/java/org/apache/hudi/utils/TestViewStorageProperties.java
##########
@@ -45,11 +45,11 @@ void testReadWriteProperties() throws IOException {
         .withStorageType(FileSystemViewStorageType.SPILLABLE_DISK)
         .withRemoteServerHost("host1")
         .withRemoteServerPort(1234).build();
-    ViewStorageProperties.createProperties(basePath, config);
-    ViewStorageProperties.createProperties(basePath, config);
-    ViewStorageProperties.createProperties(basePath, config);
+    ViewStorageProperties.createProperties(basePath, config, null);
+    ViewStorageProperties.createProperties(basePath, config, null);
+    ViewStorageProperties.createProperties(basePath, config, null);
 
-    FileSystemViewStorageConfig readConfig = ViewStorageProperties.loadFromProperties(basePath);
+    FileSystemViewStorageConfig readConfig = ViewStorageProperties.loadFromProperties(basePath, null);

Review comment:
       ditto.

##########
File path: hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java
##########
@@ -106,7 +106,7 @@ void testInstantTimeDiff() {
   void testDumpRemoteViewStorageConfig() throws IOException {
     Configuration conf = TestConfigurations.getDefaultConf(tempFile.getAbsolutePath());
     StreamerUtil.createWriteClient(conf);
-    FileSystemViewStorageConfig storageConfig = ViewStorageProperties.loadFromProperties(conf.getString(FlinkOptions.PATH));
+    FileSystemViewStorageConfig storageConfig = ViewStorageProperties.loadFromProperties(conf.getString(FlinkOptions.PATH), null);
     assertThat(storageConfig.getStorageType(), is(FileSystemViewStorageType.REMOTE_FIRST));

Review comment:
       Use `new Configuration` to avoid null.

##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/ViewStorageProperties.java
##########
@@ -61,10 +63,10 @@ public static void createProperties(
   /**
    * Read the {@link FileSystemViewStorageConfig} with given table base path.
    */
-  public static FileSystemViewStorageConfig loadFromProperties(String basePath) {
+  public static FileSystemViewStorageConfig loadFromProperties(String basePath, Configuration configuration) {
     Path propertyPath = getPropertiesFilePath(basePath);
     LOG.info("Loading filesystem view storage properties from " + propertyPath);

Review comment:
       `configuration` => `conf`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org