You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/14 23:55:37 UTC
[GitHub] [hudi] rmahindra123 opened a new pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
rmahindra123 opened a new pull request #3660:
URL: https://github.com/apache/hudi/pull/3660
Implement and test Hive Sync support for Kafka Connect. After every END_COMMIT, the coordinator centrally schedules a hive meta store sync. Currently, blocked on runtime dependency issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-971015764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970839716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua edited a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua edited a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well, with Hive 2.3.9:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua edited a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua edited a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well, with Hive 2.3.9:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970809735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
rmahindra123 commented on a change in pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#discussion_r755452612
##########
File path: hudi-kafka-connect/README.md
##########
@@ -56,30 +50,57 @@ After building the package, we need to install the Apache Kafka
### 1 - Starting the environment
-To try out the Connect Sink locally, set up a Kafka broker locally. Download the latest apache kafka from https://kafka.apache.org/downloads.
-Once downloaded and built, run the Zookeeper server and Kafka server using the command line tools.
+For runtime dependencies, we encourage using the confluent HDFS connector jars. We have tested our setup with version `10.1.0`.
+After downloading the connector, copy the jars from the lib folder to the Kafka Connect classpath.
```bash
-export KAFKA_HOME=/path/to/kafka_install_dir
-cd $KAFKA_KAFKA_HOME
-./bin/zookeeper-server-start.sh ./config/zookeeper.properties
-./bin/kafka-server-start.sh ./config/server.properties
+confluent-hub install confluentinc/kafka-connect-hdfs:10.1.0
+cp confluentinc-kafka-connect-hdfs-10.1.0/lib/*.jars /usr/local/share/java/hudi-kafka-connect/
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#discussion_r755589798
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/utils/KafkaConnectUtils.java
##########
@@ -85,9 +87,15 @@ public static int getLatestNumPartitions(String bootstrapServers, String topicNa
*
* @return
*/
- public static Configuration getDefaultHadoopConf() {
+ public static Configuration getDefaultHadoopConf(KafkaConnectConfigs connectConfigs) {
Configuration hadoopConf = new Configuration();
- hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
+ connectConfigs.getProps().keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to hadoop/hive configs
+ return !prop.toString().startsWith(HOODIE_CONF_PREFIX);
+ }).forEach(prop -> {
+ hadoopConf.set(prop.toString(), connectConfigs.getProps().get(prop.toString()).toString());
+ });
Review comment:
Got it. Then let's keep this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970834854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-961588399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-977112409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
}, {
"hash" : "0b41218802fa9dddc4154129a04079f4037204e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650",
"triggerID" : "0b41218802fa9dddc4154129a04079f4037204e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
* 0b41218802fa9dddc4154129a04079f4037204e2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
rmahindra123 commented on a change in pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#discussion_r755465220
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/utils/KafkaConnectUtils.java
##########
@@ -85,9 +87,15 @@ public static int getLatestNumPartitions(String bootstrapServers, String topicNa
*
* @return
*/
- public static Configuration getDefaultHadoopConf() {
+ public static Configuration getDefaultHadoopConf(KafkaConnectConfigs connectConfigs) {
Configuration hadoopConf = new Configuration();
- hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
+ connectConfigs.getProps().keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to hadoop/hive configs
+ return !prop.toString().startsWith(HOODIE_CONF_PREFIX);
+ }).forEach(prop -> {
+ hadoopConf.set(prop.toString(), connectConfigs.getProps().get(prop.toString()).toString());
+ });
Review comment:
I could, but i have to convert the kafka map configs to hadoop configs. also later we can change this logic if required. for instance, if we want to just have hadoop confs start with "conf.hadoop." in kafka connect, we can do that independently. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970814884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970809735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970918622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-977159586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
}, {
"hash" : "0b41218802fa9dddc4154129a04079f4037204e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650",
"triggerID" : "0b41218802fa9dddc4154129a04079f4037204e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0b41218802fa9dddc4154129a04079f4037204e2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#discussion_r754693025
##########
File path: hudi-kafka-connect/README.md
##########
@@ -56,30 +50,57 @@ After building the package, we need to install the Apache Kafka
### 1 - Starting the environment
-To try out the Connect Sink locally, set up a Kafka broker locally. Download the latest apache kafka from https://kafka.apache.org/downloads.
-Once downloaded and built, run the Zookeeper server and Kafka server using the command line tools.
+For runtime dependencies, we encourage using the confluent HDFS connector jars. We have tested our setup with version `10.1.0`.
+After downloading the connector, copy the jars from the lib folder to the Kafka Connect classpath.
```bash
-export KAFKA_HOME=/path/to/kafka_install_dir
-cd $KAFKA_KAFKA_HOME
-./bin/zookeeper-server-start.sh ./config/zookeeper.properties
-./bin/kafka-server-start.sh ./config/server.properties
+confluent-hub install confluentinc/kafka-connect-hdfs:10.1.0
+cp confluentinc-kafka-connect-hdfs-10.1.0/lib/*.jars /usr/local/share/java/hudi-kafka-connect/
Review comment:
Should the HDFS connector lib path `confluentinc-kafka-connect-hdfs-10.1.0/lib` be added to `plugin.path` instead of copying the jars like this?
##########
File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/utils/KafkaConnectUtils.java
##########
@@ -85,9 +87,15 @@ public static int getLatestNumPartitions(String bootstrapServers, String topicNa
*
* @return
*/
- public static Configuration getDefaultHadoopConf() {
+ public static Configuration getDefaultHadoopConf(KafkaConnectConfigs connectConfigs) {
Configuration hadoopConf = new Configuration();
- hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
+ connectConfigs.getProps().keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to hadoop/hive configs
+ return !prop.toString().startsWith(HOODIE_CONF_PREFIX);
+ }).forEach(prop -> {
+ hadoopConf.set(prop.toString(), connectConfigs.getProps().get(prop.toString()).toString());
+ });
Review comment:
I see similar code in `KafkaOffsetGen::excludeHoodieConfigs()`. Could you keep one of them?
##########
File path: hudi-kafka-connect/README.md
##########
@@ -56,30 +50,57 @@ After building the package, we need to install the Apache Kafka
### 1 - Starting the environment
-To try out the Connect Sink locally, set up a Kafka broker locally. Download the latest apache kafka from https://kafka.apache.org/downloads.
-Once downloaded and built, run the Zookeeper server and Kafka server using the command line tools.
+For runtime dependencies, we encourage using the confluent HDFS connector jars. We have tested our setup with version `10.1.0`.
+After downloading the connector, copy the jars from the lib folder to the Kafka Connect classpath.
```bash
-export KAFKA_HOME=/path/to/kafka_install_dir
-cd $KAFKA_KAFKA_HOME
-./bin/zookeeper-server-start.sh ./config/zookeeper.properties
-./bin/kafka-server-start.sh ./config/server.properties
+confluent-hub install confluentinc/kafka-connect-hdfs:10.1.0
+cp confluentinc-kafka-connect-hdfs-10.1.0/lib/*.jars /usr/local/share/java/hudi-kafka-connect/
```
-Wait until the kafka cluster is up and running.
+### 2 - Set up the docker containers
-### 2 - Set up the schema registry
+To run the connect locally, we need kafka, zookeeper, hdfs, hive etc. To make the setup easier, we use the docker
+containers from the hudi docker demo. Refer to [this link for the setup](https://hudi.apache.org/docs/docker_demo)
+
+Essentially, follow the steps listed here:
+
+/etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts
+```bash
+127.0.0.1 adhoc-1
+127.0.0.1 adhoc-2
+127.0.0.1 namenode
+127.0.0.1 datanode1
+127.0.0.1 hiveserver
+127.0.0.1 hivemetastore
+127.0.0.1 kafkabroker
+127.0.0.1 sparkmaster
+127.0.0.1 zookeeper
+```
+
+Bring up the docker containers
+```bash
+cd $HUDI_DIR/docker
+./setup_demo.sh
+```
+
+The schema registry and kafka connector can be run from host system directly (mac/ linux).
+
+### 3 - Set up the schema registry
Hudi leverages schema registry to obtain the latest schema when writing records. While it supports most popular schema
registries, we use Confluent schema registry. Download the latest confluent platform and run the schema registry
-service.
+service.
+
+NOTE: You might need to change the port from `8081` to `8082`.
```bash
cd $CONFLUENT_DIR
+/bin/kafka-configs --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact
./bin/schema-registry-start etc/schema-registry/schema-registry.properties
Review comment:
As a follow-up, maybe we can add all these Kafka-related environment setup, including schema registry, to the docker demo, to make it easier for users to try out.
##########
File path: hudi-kafka-connect/demo/config-sink.json
##########
@@ -1,19 +1,31 @@
{
"name": "hudi-sink",
"config": {
- "bootstrap.servers": "localhost:9092",
+ "bootstrap.servers": "kafkabroker:9092",
"connector.class": "org.apache.hudi.connect.HoodieSinkConnector",
- "tasks.max": "4",
Review comment:
Can this be kept as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-977108444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
}, {
"hash" : "0b41218802fa9dddc4154129a04079f4037204e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b41218802fa9dddc4154129a04079f4037204e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
* 0b41218802fa9dddc4154129a04079f4037204e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-977108444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
}, {
"hash" : "0b41218802fa9dddc4154129a04079f4037204e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b41218802fa9dddc4154129a04079f4037204e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
* 0b41218802fa9dddc4154129a04079f4037204e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970918622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-961588399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua edited a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua edited a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well, with Hive 2.3.9:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970814884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970839716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-971015764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-977112409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
}, {
"hash" : "0b41218802fa9dddc4154129a04079f4037204e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650",
"triggerID" : "0b41218802fa9dddc4154129a04079f4037204e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b3e9252002f9514d49c152920dd88dae26d771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3417)
* 0b41218802fa9dddc4154129a04079f4037204e2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-961588399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-962064568
I hit the runtime dependency issue as well:
```
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
```
Full stacktrace here: https://gist.github.com/yihua/c108a9d04ce35158f6603a07ef4c21aa
@codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3660: [WIP] [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-919592520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
rmahindra123 commented on a change in pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#discussion_r755452809
##########
File path: hudi-kafka-connect/README.md
##########
@@ -56,30 +50,57 @@ After building the package, we need to install the Apache Kafka
### 1 - Starting the environment
-To try out the Connect Sink locally, set up a Kafka broker locally. Download the latest apache kafka from https://kafka.apache.org/downloads.
-Once downloaded and built, run the Zookeeper server and Kafka server using the command line tools.
+For runtime dependencies, we encourage using the confluent HDFS connector jars. We have tested our setup with version `10.1.0`.
+After downloading the connector, copy the jars from the lib folder to the Kafka Connect classpath.
```bash
-export KAFKA_HOME=/path/to/kafka_install_dir
-cd $KAFKA_KAFKA_HOME
-./bin/zookeeper-server-start.sh ./config/zookeeper.properties
-./bin/kafka-server-start.sh ./config/server.properties
+confluent-hub install confluentinc/kafka-connect-hdfs:10.1.0
+cp confluentinc-kafka-connect-hdfs-10.1.0/lib/*.jars /usr/local/share/java/hudi-kafka-connect/
```
-Wait until the kafka cluster is up and running.
+### 2 - Set up the docker containers
-### 2 - Set up the schema registry
+To run the connect locally, we need kafka, zookeeper, hdfs, hive etc. To make the setup easier, we use the docker
+containers from the hudi docker demo. Refer to [this link for the setup](https://hudi.apache.org/docs/docker_demo)
+
+Essentially, follow the steps listed here:
+
+/etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts
+```bash
+127.0.0.1 adhoc-1
+127.0.0.1 adhoc-2
+127.0.0.1 namenode
+127.0.0.1 datanode1
+127.0.0.1 hiveserver
+127.0.0.1 hivemetastore
+127.0.0.1 kafkabroker
+127.0.0.1 sparkmaster
+127.0.0.1 zookeeper
+```
+
+Bring up the docker containers
+```bash
+cd $HUDI_DIR/docker
+./setup_demo.sh
+```
+
+The schema registry and kafka connector can be run from host system directly (mac/ linux).
+
+### 3 - Set up the schema registry
Hudi leverages schema registry to obtain the latest schema when writing records. While it supports most popular schema
registries, we use Confluent schema registry. Download the latest confluent platform and run the schema registry
-service.
+service.
+
+NOTE: You might need to change the port from `8081` to `8082`.
```bash
cd $CONFLUENT_DIR
+/bin/kafka-configs --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact
./bin/schema-registry-start etc/schema-registry/schema-registry.properties
Review comment:
ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #3660:
URL: https://github.com/apache/hudi/pull/3660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-961588399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3660: [HUDI-2325] Add hive sync support to kafka connect
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3660:
URL: https://github.com/apache/hudi/pull/3660#issuecomment-970834854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218",
"triggerID" : "5dbb05deb9889ef9814e1d9f731b23741aa254b0",
"triggerType" : "PUSH"
}, {
"hash" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415",
"triggerID" : "f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb",
"triggerType" : "PUSH"
}, {
"hash" : "49b3e9252002f9514d49c152920dd88dae26d771",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b3e9252002f9514d49c152920dd88dae26d771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5dbb05deb9889ef9814e1d9f731b23741aa254b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2218)
* f3b7ebdd26e0ec8a21ad6254a1f4b5578f9523bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3415)
* 49b3e9252002f9514d49c152920dd88dae26d771 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org