You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhuanshenbsj1 (via GitHub)" <gi...@apache.org> on 2024/09/23 16:48:58 UTC

[PR] [flink] Optimize Hadoop configuration loading [paimon]

zhuanshenbsj1 opened a new pull request, #4241:
URL: https://github.com/apache/paimon/pull/4241

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   1.The HiveCatalog reuses the HadoopUtils.addHadoopConfIfFound function to initialize the HDFS configuration, and adjust the configuration files being loaded.
   
   - Before the modification: 
   ```
     - HiveCatalog will loadingļ¼š hdfs-site.xml, core-site.xml
     - CatalogContext will loading:  hdfs-site.xml, core-site.xml, yarn-site.xml,mapred-site.xml
   ```
   - After the modification:
   `-  hdfs-site.xml, core-site.xml, mapred-site.xml`
   
   We've removed the yarn-site.xml as we don't need to load the yarn configuration, but we still require the mapred-site.xml for compression configuration.
   
   2.Adjusted priority for Hadoop configuration loading priority from low to high is as follows
   - Before the modification: 
   ```
     - HADOOP_HOME
     - Paimon catalog or paimon table configuration(hadoop-conf-dir)
     - HADOOP_CONF_DIR
     - paimon table advanced configurations
   ```
   - After the modification: 
   ```
     - HADOOP_HOME
     - HADOOP_CONF_DIR
     - Paimon catalog or paimon table configuration(hadoop-conf-dir)
     - paimon table advanced configurations
   ```
   
   
   
   ### Tests
   
   HadoopConfigLoadingTest.loadOverlappingConfig
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [flink] Optimize Hadoop configuration loading [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #4241: [flink] Optimize Hadoop configuration loading
URL: https://github.com/apache/paimon/pull/4241


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [flink] Optimize Hadoop configuration loading [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4241:
URL: https://github.com/apache/paimon/pull/4241


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org