You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/09 08:12:42 UTC
[GitHub] [iceberg] coolderli opened a new pull request #2799: Flink: merge hiveConf to hiveCatalog
coolderli opened a new pull request #2799:
URL: https://github.com/apache/iceberg/pull/2799
Merge the configurations from hive-conf-dir to the current Hadoop configuration. And users can use HiveCatalog to load iceberg tables.
```
CatalogLoader loader = CatalogLoader.hive("iceberg_catalog", new Configuration(), properties);
TableLoader tableLoader = TableLoader.fromCatalog(loader, TableIdentifier.of("db", "table"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coolderli commented on a change in pull request #2799: Flink: merge the configurations of hive-conf-dir to CatalogLoader
Posted by GitBox <gi...@apache.org>.
coolderli commented on a change in pull request #2799:
URL: https://github.com/apache/iceberg/pull/2799#discussion_r667590594
##########
File path: flink/src/main/java/org/apache/iceberg/flink/CatalogLoader.java
##########
@@ -51,7 +51,9 @@ static CatalogLoader hadoop(String name, Configuration hadoopConf, Map<String, S
}
static CatalogLoader hive(String name, Configuration hadoopConf, Map<String, String> properties) {
- return new HiveCatalogLoader(name, hadoopConf, properties);
+ String hiveConfDir = properties.get(FlinkCatalogFactory.HIVE_CONF_DIR);
+ Configuration newHadoopConf = FlinkCatalogFactory.mergeHiveConf(hadoopConf, hiveConfDir);
Review comment:
Yes, it can be loaded into `HiveCatalog` successfully. But there are some other configurations (For example `hadoop.security.authentication`，`hive.metastore.kerberos.principal`) that could not be load by properties. Of course, this can be passed in through the `hadoopConf` . However, users must always pay attention to the consistency of code configuration and cluster environment(the configuration of hive-site.xml). It is true that users can pass the hive-site.xml to `hadoopConf` outside this method, but this needs to be consistent with hive-site.xml in hivecatalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #2799: Flink: merge the configurations of hive-conf-dir to CatalogLoader
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #2799:
URL: https://github.com/apache/iceberg/pull/2799#discussion_r667571844
##########
File path: flink/src/main/java/org/apache/iceberg/flink/CatalogLoader.java
##########
@@ -51,7 +51,9 @@ static CatalogLoader hadoop(String name, Configuration hadoopConf, Map<String, S
}
static CatalogLoader hive(String name, Configuration hadoopConf, Map<String, String> properties) {
- return new HiveCatalogLoader(name, hadoopConf, properties);
+ String hiveConfDir = properties.get(FlinkCatalogFactory.HIVE_CONF_DIR);
+ Configuration newHadoopConf = FlinkCatalogFactory.mergeHiveConf(hadoopConf, hiveConfDir);
Review comment:
You are using flink datastream API to submit the flink jobs right ? In theory, the `hadoopConf` cloud be passed as any hadoop `Configuration` if your want (For example, you can add hive-site.xml to `hadoopConf` outside this method). Another side, we could load the `HiveCatalog` without specifying any hive-site.xml configurations because we will set all the necessary key-values configurations from `propeties` to `hiveConf` ( see [code](https://github.com/apache/iceberg/blob/90225d6c9413016d611e2ce5eff37db1bc1b4fc5/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L96)). So all you need to do is filing the correct hive properties in the `Map<String, String> properties`:
```java
Map<String, String> properties = Maps.newHashMap();
properties.set("uri", "thrift://localhost:9083");
properties.set("warehouse", "hdfs://nn:9090/path/to/warehouse");
properties.set("clients", "5");
```
Then you cloud load the correct `HiveCatalog`. So I think we don't need to have this changes in our master codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org