You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/03 14:05:26 UTC

[GitHub] [hudi] xushiyan opened a new pull request, #7131: [HUDI-2673] Add kafka connect bundle to validation test

xushiyan opened a new pull request, #7131:
URL: https://github.com/apache/hudi/pull/7131

   ### Change Logs
   
   Add hudi-kafka-connect-bundle to bundle validation test.
   
   ### Impact
   
   NA
   
   ### Risk level
   
   none
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on a diff in pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #7131:
URL: https://github.com/apache/hudi/pull/7131#discussion_r1014483789


##########
packaging/bundle-validation/Dockerfile-base:
##########
@@ -47,3 +48,15 @@ RUN wget https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK
     && tar -xf $WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION.tgz -C $WORKDIR/ \
     && rm $WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION.tgz
 ENV SPARK_HOME=$WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION
+
+RUN wget https://packages.confluent.io/archive/${CONFLUENT_VERSION%.*}/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz -P "$WORKDIR" \
+    && tar -xf $WORKDIR/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz -C $WORKDIR/ \
+    && rm $WORKDIR/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz
+ENV CONFLUENT_HOME=$WORKDIR/confluent-$CONFLUENT_VERSION
+
+RUN wget https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-hdfs/versions/10.1.13/confluentinc-kafka-connect-hdfs-10.1.13.zip -P "$WORKDIR" \

Review Comment:
   Do we want to parameterize the version of `kafka-connect-hdfs` as well?  We may want to upgrade it in the future.



##########
hudi-kafka-connect/demo/setupKafka.sh:
##########
@@ -65,7 +64,7 @@ numBatch=1
 recordValue=0
 recreateTopic="Y"
 
-while getopts ":n:b:tf:k:m:r:o:l:p:s:-:" opt; do
+while getopts ":n:b:t:f:k:m:r:o:l:p:s:-:" opt; do

Review Comment:
   `-t` does not expect an argument so we should have `t` instead `t:`.  This change should be reverted.



##########
packaging/bundle-validation/validate.sh:
##########
@@ -112,10 +113,41 @@ test_utilities_bundle () {
 }
 
 
+##
+# Function to test the kafka-connect bundle.
+# It runs zookeeper, kafka broker, schema registry, and connector worker.
+# After producing and consuming data, it checks successful commit under `.hoodie/`
+#
+# 1st arg: path to the hudi-kafka-connect-bundle.jar (for writing data)
+#
+# env vars (defined in container):
+#   CONFLUENT_HOME: path to the confluent community directory
+#   KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH: path to install hudi-kafka-connect-bundle.jar
+##
+test_kafka_connect_bundle() {
+    KAFKA_CONNECT_JAR=$1
+    cp $KAFKA_CONNECT_JAR $KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH
+    $CONFLUENT_HOME/bin/zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper.properties &
+    $CONFLUENT_HOME/bin/kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties &
+    sleep 10
+    $CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties &

Review Comment:
   Kill these background processes at the end?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1305940713

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     }, {
       "hash" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797",
       "triggerID" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "96a2683069ea15e8026ec8a25871b2ce8f96ebc9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12859",
       "triggerID" : "96a2683069ea15e8026ec8a25871b2ce8f96ebc9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 300ffe6e2400da72208bf9b759e004b3b06af02b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797) 
   * 96a2683069ea15e8026ec8a25871b2ce8f96ebc9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12859) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1304417008

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     }, {
       "hash" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5cb83da5cd9b128c8c59ca7adc857f407c10495 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789) 
   * 300ffe6e2400da72208bf9b759e004b3b06af02b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1302193154

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a6092baa5c8be038ff0ca6b016ae911100401382 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764) 
   * b096bd91fbe795114f6c2f4540a0a554d08f774f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1302184681

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a6092baa5c8be038ff0ca6b016ae911100401382 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1303559801

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5cb83da5cd9b128c8c59ca7adc857f407c10495 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on a diff in pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #7131:
URL: https://github.com/apache/hudi/pull/7131#discussion_r1014574815


##########
packaging/bundle-validation/validate.sh:
##########
@@ -112,10 +113,41 @@ test_utilities_bundle () {
 }
 
 
+##
+# Function to test the kafka-connect bundle.
+# It runs zookeeper, kafka broker, schema registry, and connector worker.
+# After producing and consuming data, it checks successful commit under `.hoodie/`
+#
+# 1st arg: path to the hudi-kafka-connect-bundle.jar (for writing data)
+#
+# env vars (defined in container):
+#   CONFLUENT_HOME: path to the confluent community directory
+#   KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH: path to install hudi-kafka-connect-bundle.jar
+##
+test_kafka_connect_bundle() {
+    KAFKA_CONNECT_JAR=$1
+    cp $KAFKA_CONNECT_JAR $KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH
+    $CONFLUENT_HOME/bin/zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper.properties &
+    $CONFLUENT_HOME/bin/kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties &
+    sleep 10
+    $CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties &

Review Comment:
   the container exits will end all these



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1303439366

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b096bd91fbe795114f6c2f4540a0a554d08f774f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766) 
   * b5cb83da5cd9b128c8c59ca7adc857f407c10495 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1302285600

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a6092baa5c8be038ff0ca6b016ae911100401382 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764) 
   * b096bd91fbe795114f6c2f4540a0a554d08f774f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1304425389

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     }, {
       "hash" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797",
       "triggerID" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5cb83da5cd9b128c8c59ca7adc857f407c10495 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789) 
   * 300ffe6e2400da72208bf9b759e004b3b06af02b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on a diff in pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #7131:
URL: https://github.com/apache/hudi/pull/7131#discussion_r1014586450


##########
hudi-kafka-connect/demo/setupKafka.sh:
##########
@@ -65,7 +64,7 @@ numBatch=1
 recordValue=0
 recreateTopic="Y"
 
-while getopts ":n:b:tf:k:m:r:o:l:p:s:-:" opt; do
+while getopts ":n:b:t:f:k:m:r:o:l:p:s:-:" opt; do

Review Comment:
   fixed



##########
packaging/bundle-validation/Dockerfile-base:
##########
@@ -47,3 +48,15 @@ RUN wget https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK
     && tar -xf $WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION.tgz -C $WORKDIR/ \
     && rm $WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION.tgz
 ENV SPARK_HOME=$WORKDIR/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION
+
+RUN wget https://packages.confluent.io/archive/${CONFLUENT_VERSION%.*}/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz -P "$WORKDIR" \
+    && tar -xf $WORKDIR/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz -C $WORKDIR/ \
+    && rm $WORKDIR/confluent-community-$CONFLUENT_VERSION-2.12.tar.gz
+ENV CONFLUENT_HOME=$WORKDIR/confluent-$CONFLUENT_VERSION
+
+RUN wget https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-hdfs/versions/10.1.13/confluentinc-kafka-connect-hdfs-10.1.13.zip -P "$WORKDIR" \

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1303451513

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b096bd91fbe795114f6c2f4540a0a554d08f774f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766) 
   * b5cb83da5cd9b128c8c59ca7adc857f407c10495 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1305933437

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     }, {
       "hash" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797",
       "triggerID" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "96a2683069ea15e8026ec8a25871b2ce8f96ebc9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "96a2683069ea15e8026ec8a25871b2ce8f96ebc9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 300ffe6e2400da72208bf9b759e004b3b06af02b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797) 
   * 96a2683069ea15e8026ec8a25871b2ce8f96ebc9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua merged pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
yihua merged PR #7131:
URL: https://github.com/apache/hudi/pull/7131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1304438534

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12789",
       "triggerID" : "b5cb83da5cd9b128c8c59ca7adc857f407c10495",
       "triggerType" : "PUSH"
     }, {
       "hash" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797",
       "triggerID" : "300ffe6e2400da72208bf9b759e004b3b06af02b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 300ffe6e2400da72208bf9b759e004b3b06af02b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12797) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on a diff in pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #7131:
URL: https://github.com/apache/hudi/pull/7131#discussion_r1015752846


##########
packaging/bundle-validation/validate.sh:
##########
@@ -112,10 +113,41 @@ test_utilities_bundle () {
 }
 
 
+##
+# Function to test the kafka-connect bundle.
+# It runs zookeeper, kafka broker, schema registry, and connector worker.
+# After producing and consuming data, it checks successful commit under `.hoodie/`
+#
+# 1st arg: path to the hudi-kafka-connect-bundle.jar (for writing data)
+#
+# env vars (defined in container):
+#   CONFLUENT_HOME: path to the confluent community directory
+#   KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH: path to install hudi-kafka-connect-bundle.jar
+##
+test_kafka_connect_bundle() {
+    KAFKA_CONNECT_JAR=$1
+    cp $KAFKA_CONNECT_JAR $KAFKA_CONNECT_PLUGIN_PATH_LIB_PATH
+    $CONFLUENT_HOME/bin/zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper.properties &
+    $CONFLUENT_HOME/bin/kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties &
+    sleep 10
+    $CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties &

Review Comment:
   It's okay for CI.  If I'd like to run this script locally for validation, the processes must be killed.  This can be a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1302301612

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12764",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766",
       "triggerID" : "b096bd91fbe795114f6c2f4540a0a554d08f774f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b096bd91fbe795114f6c2f4540a0a554d08f774f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12766) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on a diff in pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #7131:
URL: https://github.com/apache/hudi/pull/7131#discussion_r1012951236


##########
packaging/bundle-validation/validate.sh:
##########
@@ -135,3 +167,15 @@ if [ "$?" -ne 0 ]; then
     exit 1
 fi
 echo "::warning::validate.sh done testing utilities slim bundle"
+
+if [[  $SPARK_HOME == *"spark-3.1"* ]]

Review Comment:
   gonna remove this check after all base images uploaded. we should test any artifacts produced by any profile



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7131: [HUDI-2673] Add kafka connect bundle to validation test

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7131:
URL: https://github.com/apache/hudi/pull/7131#issuecomment-1302176380

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a6092baa5c8be038ff0ca6b016ae911100401382",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a6092baa5c8be038ff0ca6b016ae911100401382 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org