You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yik San Chan <ev...@gmail.com> on 2021/04/26 09:34:27 UTC
Flink Hive connector: hive-conf-dir supports hdfs URI, while
hadoop-conf-dir supports local path only?
Hi community,
This question is cross-posted on Stack Overflow
https://stackoverflow.com/questions/67264156/flink-hive-connector-hive-conf-dir-supports-hdfs-uri-while-hadoop-conf-dir-sup
In my current setup, local dev env can access testing env. I would like to
run Flink job on local dev env, while reading/writing data from/to testing
env Hive.
This is what I do:
```
CREATE CATALOG hive WITH (
'type' = 'hive',
'hive-conf-dir' = 'hdfs://testhdp273/hive/conf'
)
```
However, I realizes I also need to specify a matching Hadoop classpath,
therefore I want to also define `hadoop-conf-dir` that actually points to
the hadoop classpath in testing env. However, as said in [docs](
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/hive/#hadoop-conf-dir
):
> Path to Hadoop conf dir. Only local file system paths are supported. The
recommended way to set Hadoop conf is via the HADOOP_CONF_DIR environment
variable. Use the option only if environment variable doesn't work for you,
e.g. if you want to configure each HiveCatalog separately.
I wonder why hadoop-conf-dir only supports local path, while hive-conf-dir
supports any legit hdfs path? Any work around to this problem?
Any help? Thanks!
Best,
Yik San