You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/28 07:51:13 UTC
[GitHub] [hudi] cdmikechen opened a new pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
cdmikechen opened a new pull request #4458:
URL: https://github.com/apache/hudi/pull/4458
## What is the purpose of the pull request
KafkaConnect use `org.apache.hudi.DataSourceUtils` to build HiveSyncConfig now, but `DataSourceUtils` import some spark dependencies. So that Hive sync will fail because of the application of related classes.
```
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.types.DataType
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 66 more
```
## Brief change log
- Add a separate method to create HiveSyncConfig
## Verify this pull request
Need to add Hive sync test by https://issues.apache.org/jira/browse/HUDI-2673
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [ ] CI is green
- [x] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001922806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#discussion_r779920542
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java
##########
@@ -185,20 +187,50 @@ private void syncMeta() {
}
private void syncHive() {
- HiveSyncConfig hiveSyncConfig = DataSourceUtils.buildHiveSyncConfig(
- new TypedProperties(connectConfigs.getProps()),
- tableBasePath,
- "PARQUET");
+ HiveSyncConfig hiveSyncConfig = buildSyncConfig(new TypedProperties(connectConfigs.getProps()), tableBasePath);
+ String url;
+ if (!StringUtils.isNullOrEmpty(hiveSyncConfig.syncMode) && HiveSyncMode.of(hiveSyncConfig.syncMode) == HiveSyncMode.HMS) {
+ url = hadoopConf.get(KafkaConnectConfigs.HIVE_METASTORE_URIS);
+ } else {
+ url = hiveSyncConfig.jdbcUrl;
+ }
+
LOG.info("Syncing target hoodie table with hive table("
+ hiveSyncConfig.tableName
+ "). Hive metastore URL :"
- + hiveSyncConfig.jdbcUrl
+ + url
+ ", basePath :" + tableBasePath);
- LOG.info("Hive Sync Conf => " + hiveSyncConfig.toString());
+ LOG.info("Hive Sync Conf => " + hiveSyncConfig);
FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConf);
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
LOG.info("Hive Conf => " + hiveConf.getAllProperties().toString());
new HiveSyncTool(hiveSyncConfig, hiveConf, fs).syncHoodieTable();
}
+
+ /**
+ * Build Hive Sync Config
+ */
+ public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {
Review comment:
If the problem is due to irrelevant classes to Kafka Connect imported from `DataSourceUtils`, is it possible to move `DataSourceUtils::buildHiveSyncConfig` to a different/new util class so buildHiveSyncConfig() can still be reused here, instead of duplicating the code in hudi-kafka-connect module? Should that solve the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008263265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
}, {
"hash" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027",
"triggerID" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
* 8d1205d795799594aefe852a6ddeeb13fb38072f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008262905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
}, {
"hash" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
* 8d1205d795799594aefe852a6ddeeb13fb38072f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#discussion_r780742854
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java
##########
@@ -185,20 +187,50 @@ private void syncMeta() {
}
private void syncHive() {
- HiveSyncConfig hiveSyncConfig = DataSourceUtils.buildHiveSyncConfig(
- new TypedProperties(connectConfigs.getProps()),
- tableBasePath,
- "PARQUET");
+ HiveSyncConfig hiveSyncConfig = buildSyncConfig(new TypedProperties(connectConfigs.getProps()), tableBasePath);
+ String url;
+ if (!StringUtils.isNullOrEmpty(hiveSyncConfig.syncMode) && HiveSyncMode.of(hiveSyncConfig.syncMode) == HiveSyncMode.HMS) {
+ url = hadoopConf.get(KafkaConnectConfigs.HIVE_METASTORE_URIS);
+ } else {
+ url = hiveSyncConfig.jdbcUrl;
+ }
+
LOG.info("Syncing target hoodie table with hive table("
+ hiveSyncConfig.tableName
+ "). Hive metastore URL :"
- + hiveSyncConfig.jdbcUrl
+ + url
+ ", basePath :" + tableBasePath);
- LOG.info("Hive Sync Conf => " + hiveSyncConfig.toString());
+ LOG.info("Hive Sync Conf => " + hiveSyncConfig);
FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConf);
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
LOG.info("Hive Conf => " + hiveConf.getAllProperties().toString());
new HiveSyncTool(hiveSyncConfig, hiveConf, fs).syncHoodieTable();
}
+
+ /**
+ * Build Hive Sync Config
+ */
+ public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {
Review comment:
Let's move this util method to `KafkaConnectUtils` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008262905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
}, {
"hash" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
* 8d1205d795799594aefe852a6ddeeb13fb38072f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001923904
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001922806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001957335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#discussion_r780742608
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java
##########
@@ -185,20 +187,50 @@ private void syncMeta() {
}
private void syncHive() {
- HiveSyncConfig hiveSyncConfig = DataSourceUtils.buildHiveSyncConfig(
- new TypedProperties(connectConfigs.getProps()),
- tableBasePath,
- "PARQUET");
+ HiveSyncConfig hiveSyncConfig = buildSyncConfig(new TypedProperties(connectConfigs.getProps()), tableBasePath);
+ String url;
+ if (!StringUtils.isNullOrEmpty(hiveSyncConfig.syncMode) && HiveSyncMode.of(hiveSyncConfig.syncMode) == HiveSyncMode.HMS) {
+ url = hadoopConf.get(KafkaConnectConfigs.HIVE_METASTORE_URIS);
+ } else {
+ url = hiveSyncConfig.jdbcUrl;
+ }
+
LOG.info("Syncing target hoodie table with hive table("
+ hiveSyncConfig.tableName
+ "). Hive metastore URL :"
- + hiveSyncConfig.jdbcUrl
+ + url
+ ", basePath :" + tableBasePath);
- LOG.info("Hive Sync Conf => " + hiveSyncConfig.toString());
+ LOG.info("Hive Sync Conf => " + hiveSyncConfig);
FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConf);
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
LOG.info("Hive Conf => " + hiveConf.getAllProperties().toString());
new HiveSyncTool(hiveSyncConfig, hiveConf, fs).syncHoodieTable();
}
+
+ /**
+ * Build Hive Sync Config
+ */
+ public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {
Review comment:
@cdmikechen Understood. I'm thinking about only moving util methods related Hive sync configs, not the Hive sync logic, to a separate Util class. The worry I have is that hive sync configs are spread into different places now and they may diverge if we forget to update all of them to be consistent.
We can keep this PR as is for now. @cdmikechen could you create a Jira ticket to track the Hive sync config unification, which will be done in a different PR in future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001957335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cdmikechen commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
cdmikechen commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1003704858
@yihua Thanks for reviewing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#discussion_r780742608
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java
##########
@@ -185,20 +187,50 @@ private void syncMeta() {
}
private void syncHive() {
- HiveSyncConfig hiveSyncConfig = DataSourceUtils.buildHiveSyncConfig(
- new TypedProperties(connectConfigs.getProps()),
- tableBasePath,
- "PARQUET");
+ HiveSyncConfig hiveSyncConfig = buildSyncConfig(new TypedProperties(connectConfigs.getProps()), tableBasePath);
+ String url;
+ if (!StringUtils.isNullOrEmpty(hiveSyncConfig.syncMode) && HiveSyncMode.of(hiveSyncConfig.syncMode) == HiveSyncMode.HMS) {
+ url = hadoopConf.get(KafkaConnectConfigs.HIVE_METASTORE_URIS);
+ } else {
+ url = hiveSyncConfig.jdbcUrl;
+ }
+
LOG.info("Syncing target hoodie table with hive table("
+ hiveSyncConfig.tableName
+ "). Hive metastore URL :"
- + hiveSyncConfig.jdbcUrl
+ + url
+ ", basePath :" + tableBasePath);
- LOG.info("Hive Sync Conf => " + hiveSyncConfig.toString());
+ LOG.info("Hive Sync Conf => " + hiveSyncConfig);
FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConf);
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
LOG.info("Hive Conf => " + hiveConf.getAllProperties().toString());
new HiveSyncTool(hiveSyncConfig, hiveConf, fs).syncHoodieTable();
}
+
+ /**
+ * Build Hive Sync Config
+ */
+ public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {
Review comment:
@cdmikechen Understood. I'm thinking about only moving util methods related Hive sync configs, not the Hive sync logic, to a separate Util class. The worry I have is that hive sync configs are spread into different places and they may diverge if we forget to update all of them to be consistent.
We can keep this PR as is for now. @cdmikechen could you create a Jira ticket to track the Hive sync config unification, which will be done in a different PR in future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cdmikechen commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
cdmikechen commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008262491
@yihua
Hi~ I've moved `buildSyncConfig` method to `KafkaConnectUtils` and open a new issue <https://issues.apache.org/jira/browse/HUDI-3199>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1001923904
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008271206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
}, {
"hash" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027",
"triggerID" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d1205d795799594aefe852a6ddeeb13fb38072f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008263265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764",
"triggerID" : "c60da50ce53a0b9ff05677f8e7243f42d39fdbd4",
"triggerType" : "PUSH"
}, {
"hash" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027",
"triggerID" : "8d1205d795799594aefe852a6ddeeb13fb38072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c60da50ce53a0b9ff05677f8e7243f42d39fdbd4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4764)
* 8d1205d795799594aefe852a6ddeeb13fb38072f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5027)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#issuecomment-1008031304
@cdmikechen @yihua : We are targeting this patch for 0.10.1. We have code freeze planned this monday. Would be nice to get this in by then. Wanted to send out a reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cdmikechen commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
cdmikechen commented on a change in pull request #4458:
URL: https://github.com/apache/hudi/pull/4458#discussion_r779957645
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java
##########
@@ -185,20 +187,50 @@ private void syncMeta() {
}
private void syncHive() {
- HiveSyncConfig hiveSyncConfig = DataSourceUtils.buildHiveSyncConfig(
- new TypedProperties(connectConfigs.getProps()),
- tableBasePath,
- "PARQUET");
+ HiveSyncConfig hiveSyncConfig = buildSyncConfig(new TypedProperties(connectConfigs.getProps()), tableBasePath);
+ String url;
+ if (!StringUtils.isNullOrEmpty(hiveSyncConfig.syncMode) && HiveSyncMode.of(hiveSyncConfig.syncMode) == HiveSyncMode.HMS) {
+ url = hadoopConf.get(KafkaConnectConfigs.HIVE_METASTORE_URIS);
+ } else {
+ url = hiveSyncConfig.jdbcUrl;
+ }
+
LOG.info("Syncing target hoodie table with hive table("
+ hiveSyncConfig.tableName
+ "). Hive metastore URL :"
- + hiveSyncConfig.jdbcUrl
+ + url
+ ", basePath :" + tableBasePath);
- LOG.info("Hive Sync Conf => " + hiveSyncConfig.toString());
+ LOG.info("Hive Sync Conf => " + hiveSyncConfig);
FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConf);
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
LOG.info("Hive Conf => " + hiveConf.getAllProperties().toString());
new HiveSyncTool(hiveSyncConfig, hiveConf, fs).syncHoodieTable();
}
+
+ /**
+ * Build Hive Sync Config
+ */
+ public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {
Review comment:
@yihua
I had checked the Hudi project when I modified the codes. Besides hive synchronization in spark, Flink also has the same problem. However, in the flink, they also redeclared a new set of variables to solve the problem.
https://github.com/apache/hudi/blob/e9efbdb63c95d2971ac67576783e122a6e271738/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java#L576-L610
Considering that if unification is a relatively large part of adjustment, it may be a better way to solve it with a new issue. Because there are some Scala logic in hive sync, it cannot be split directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #4458:
URL: https://github.com/apache/hudi/pull/4458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org