You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nylqd (via GitHub)" <gi...@apache.org> on 2023/07/24 09:05:09 UTC
[GitHub] [hudi] nylqd opened a new issue, #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
nylqd opened a new issue, #9269:
URL: https://github.com/apache/hudi/issues/9269
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
flink: 1.15, flink on yarn, submit jobgraph trough java client
hudi: 0.13.1 compile ourself with tags/0.13.1 and more logs for debug
hive: 3.1.0, kerberized
we use hudi hms catalog, and config `hive_sync.conf.dir` as local hive-site.xml folder, but it seems have not read the local hive-site.xml file
**To Reproduce**
Steps to reproduce the behavior:
1. compile tags/0.13.1 with more logs and no other changes
2. config flink sql with hive_sync.conf.dir , set to local hive conf dir include a hive-site.xml file
3. parse flink sql to job graph and submit job graph through java client to yarn cluster as per-job mode
4. find wrong hiveConf values in HiveSyncContext `e.g. hive.mestaore.kerberos.principal: hive-metastore/_HOST@EXAMPLE.COM` like same kind of default value
**Expected behavior**
hive_sync works as expected
**Environment Description**
* Hudi version : 0.13.1
* Hive version : 3.1.0 kerberized
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nylqd commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "nylqd (via GitHub)" <gi...@apache.org>.
nylqd commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1653356214
> > > hdfs path, since the code running on yarn and my hive-site.xml is in a local dir
>
> >
>
> > I guess you are right, the hive conf dir is only valid for the catalog itself, not the job, the catalog does not pass around all the hive related config options to the job.
>
> >
>
> > Maybe you can fire a fix for it, in the HoodieHiveCatalog, when generating a new catalog table, config the hive options through `hadoop.` prefix.
>
> >
>
> > Another way is to config the system variable: `HIVE_CONF_DIR`:
>
> >
>
> > 
>
>
>
> thx for ur clarification, after set hive tblproperties from hive-site.xml with prefix `hadoop.`, we finally sync schema successfully
>
>
>
> next step, we gonna try to set those properties in the HoodieHiveCatalog
After same digging, we can add all `hive-site.xml` kvs in `HoodieHiveCatalog.instantiateHiveTable()` and it works
But there are way too many properties, and we found these ones should added
- hive.metastore.uris
- hive.metastore.sasl.enabled
- hive.metastore.kerberos.principle
Should we add all properties or just add specifed ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nylqd commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "nylqd (via GitHub)" <gi...@apache.org>.
nylqd commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1659637537
>
we apply a fix as follow, you may implement a general solution for the community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1659863501
Okay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1647728384
> hdfs path, since the code running on yarn and my hive-site.xml is in a local dir
I guess you are right, the hive conf dir is only valid for the catalog itself, not the job, the catalog does not pass around all the hive related config options to the job.
Maybe you can fire a fix for it, in the HoodieHiveCatalog, when generating a new catalog table, config the hive options through `hadoop.` prefix.
Another way is to config the system variable: `HIVE_CONF_DIR`:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nylqd commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "nylqd (via GitHub)" <gi...@apache.org>.
nylqd commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1648923651
> > hdfs path, since the code running on yarn and my hive-site.xml is in a local dir
>
> I guess you are right, the hive conf dir is only valid for the catalog itself, not the job, the catalog does not pass around all the hive related config options to the job.
>
> Maybe you can fire a fix for it, in the HoodieHiveCatalog, when generating a new catalog table, config the hive options through `hadoop.` prefix.
>
> Another way is to config the system variable: `HIVE_CONF_DIR`:
>
> 
thx for ur clarification, after set hive tblproperties we finally sync schema successfully
next step, we gonna try to set those properties in the HoodieHiveCatalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9269:
URL: https://github.com/apache/hudi/issues/9269#issuecomment-1653368887
> After same digging, we can add all hive-site.xml kvs in HoodieHiveCatalog.instantiateHiveTable()
Where are these options set up into, can you show use an example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org