[PR] Add Hadoop configuration directory hadoop_conf.dir support for Flink cross-cluster Hudi read/write [hudi]

["cbg-wx (via GitHub)" <gi...@apache.org>]

cbg-wx opened a new pull request, #20283:
URL: https://github.com/apache/hudi/pull/20283

   Add hadoop_conf.dir in FlinkOptions to support for Flink cross-cluster Hudi read/write
   
   ### Describe the issue this Pull Request addresses
   
   <!-- Either describe the issue inline here with motivation behind the changes 
        (or) link to an issue by including `Closes #<issue-number>` for context. 
        If this PR includes changes to the storage format, public APIs,
        or has breaking changes, use `!` (e.g., feat!: ...) -->
   
   This pull request in order to close https://github.com/apache/hudi/issues/19191
   
   ### Summary and Changelog
   
   <!-- Short, plain-English summary of what users gain or what changed in behavior.
        Followed by a detailed log of all the changes. Highlight if any code was copied. -->
   
   1. Add hadoop_conf.dir to FlinkOptions
   
      ```java
        @AdvancedConfig
        public static final ConfigOption<String> HADOOP_CONF_DIR = ConfigOptions
             .key("hadoop_conf.dir")
             .stringType()
             .noDefaultValue()
             .withDescription("The hadoop configuration directory, where the core-site.xml, "
                     + "hdfs-site.xml etc. lie in. Used to connect to a different HDFS cluster "
                     + "for cross-cluster writes.");
      ```
   
   2. `FlinkClientUtil.java` — Add an overloaded `getHadoopConf(String hadoopConfDir)` method with `hadoop.conf.dir` support.
   
   3. `HadoopConfigurations.java` — Update getHadoopConf to support `hadoop.conf.dir`
   
      ```java
        public static org.apache.hadoop.conf.Configuration getHadoopConf(Configuration conf) {
          // Choose the base configuration source: explicit conf dir wins;
          // fall back to environment-based discovery when the option is absent
          // or the directory is invalid.
          String hadoopConfDir = conf.getString(FlinkOptions.HADOOP_CONF_DIR.key(), null);
          org.apache.hadoop.conf.Configuration hadoopConf;
          if (hadoopConfDir != null) {
            validateHadoopConfDir(hadoopConfDir);
            org.apache.hadoop.conf.Configuration dirHadoopConf= FlinkClientUtil.getHadoopConf(hadoopConfDir);
            hadoopConf = dirHadoopConf != null ? dirHadoopConf : FlinkClientUtil.getHadoopConf();
          }else {
            hadoopConf = FlinkClientUtil.getHadoopConf();
          }
          Map<String, String> options = FlinkOptions.getPropertiesWithPrefix(conf.toMap(), HADOOP_PREFIX);
          options.forEach(hadoopConf::set);
          return hadoopConf;
        }
      ```
   
      
   
   ### Impact
   
   <!-- Describe any public API or user-facing feature change or any performance impact. -->
   
   flink read/write hudi table
   
   ### Risk Level
   
   <!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
        If medium or high, explain what verification was done to mitigate the risks. -->
   
   ### Documentation Update
   
   <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
   
   - The config description must be updated if new configs are added or the default value of the configs are changed.
   - Any new feature or user-facing change requires updating the Hudi website. Please follow the 
     [[[instruction](https://hudi.apache.org/contribute/developer-setup#website)](https://hudi.apache.org/contribute/developer-setup#website)](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
   
   ### Contributor's checklist
   
   - [ ] Read through [[[contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)](https://hudi.apache.org/contribute/how-to-contribute)](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add Hadoop configuration directory hadoop_conf.dir support for Flink cross-cluster Hudi read/write [hudi]

["cbg-wx (via GitHub)" <gi...@apache.org>]

cbg-wx closed pull request #20283: Add Hadoop configuration directory hadoop_conf.dir support for Flink cross-cluster Hudi read/write
URL: https://github.com/apache/hudi/pull/20283


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org