You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/01 18:34:02 UTC

[GitHub] [hudi] wangxianghu opened a new pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

wangxianghu opened a new pull request #4930:
URL: https://github.com/apache/hudi/pull/4930


   …cess before it is transformed to DataSet
   
   ## *Tips*
   - *Thank you very much for contributing to Apache Hudi.*
   - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
   
   ## What is the purpose of the pull request
   
   Currently, we have `Transform` to transform source to target dataset before writing, but it is based on DataSet.
   In some scenarios, our kafka data is not in the right format we need, such as binlog json format.
   We need a way to extract/prepare the data we need from the original data before converting it into a DataSet.
   
   
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on a change in pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#discussion_r818363212



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -189,6 +189,11 @@ public static long totalNewMessages(OffsetRange[] ranges) {
             .defaultValue(KafkaResetOffsetStrategies.LATEST)
             .withDocumentation("Kafka consumer strategy for reading data.");
 
+    public static final ConfigProperty<String> JSON_KAFKA_PROCESSOR_CLASS_OPT = ConfigProperty
+        .key("hoodie.deltastreamer.source.kafka.json.processor.class")
+        .noDefaultValue()
+        .withDocumentation("Json kafka source pre-processor class name, used to do pre-process on kafka data.");

Review comment:
       > lets not use "pre-process" as it might be confusing. we can just say post process data after consuming from source and before giving it to deltastreamer.
   
   done
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057788742


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056352114


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057786207


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055793248


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057734795


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on a change in pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#discussion_r818383469



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/processor/JsonKafkaSourcePostProcessor.java
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.processor;
+
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Serializable;
+
+/**
+ * Base class for Json kafka source post processor. User can define their own processor that extends this class to do
+ * some post process on the incoming json string records before the records are converted to DataSet<T>.
+ */
+public abstract class JsonKafkaSourcePostProcessor implements Serializable {

Review comment:
       changed to an abstract class so as to provide a unified constructor




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057786207


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059934053


   > @wangxianghu The changes look good to me. I have a high level query. You have mentioned facing issues with binlog json format for example. Do you mean to say transformation is not possible with data in binlog json format? Data coming from binlogs also has a structure/schema assigned to it as far as I remember. Can you post a sample event where you feel this new PostProcessor you introduced might be useful? Basically I want to understand the motivation behind introducing this PR.
   
   @pratyakshsharma thanks for your review!
   {
       "database": "test",
       "table": "maxwell",
       "type": "update",
       "ts": 1449786341,
       "xid": 940786,
       "commit": true,
       "data": {"id":1, "daemon": "Firebus!  Firebus!","update_time" : "2022-02-03 12:22:42"},
       "old":  {"daemon": "Stanislaw Lem"}
     }
   
   
     > @wangxianghu The changes look good to me. I have a high level query. You have mentioned facing issues with binlog json format for example. Do you mean to say transformation is not possible with data in binlog json format? Data coming from binlogs also has a structure/schema assigned to it as far as I remember. Can you post a sample event where you feel this new PostProcessor you introduced might be useful? Basically I want to understand the motivation behind introducing this PR.
   
   It is possible to deal with data in binlog json format, but not very convenient.
   
   1. For maxwell(our company use it to capture changed data)
   ```
   {
       "database": "test",
       "table": "maxwell",
       "type": "update",
       "ts": 1449786341,
       "xid": 940786,
       "commit": true,
       "data": {"id":1, "daemon": "Firebus!  Firebus!","update_time" : "2022-02-03 12:22:42"},
       "old":  {"daemon": "Stanislaw Lem"}
     }
   ```
   all we want is just :
   
   ```
   {
       "id": 1, 
       "daemon": "Firebus!  Firebus!", 
       "update_time": "2022-02-03 12:22:42"
   }
   ```
   we can add write a processor to extract the data from the entire json and maybe do some custom define process, without configuring a huge schema file(including all the fields in the binlog json, no matter if we need them or not)
   
   2. in some scenes, we need to encode some fileds for safety purpose, the processor can help us
   
   3. sometimes our data quality is not very well, some key field let's say precombine field have null value, we can use processor to fix it
   
   4. when our schema is read from jdbc or hive, we can adjust our kafka data compatible to it. 
   
   All in all, with custom processor we can do anything we want on the incoming json data before they are converted into DataSet
   Of course `Transformer` is a very useful feature too, but it is based on Spark DataSet, and have certain requirements for data quality.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055985368


   Can you check the CI failure please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056455960


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056455960


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057853853


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057925985


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pratyakshsharma commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059936383


   Right. With Debezium, this extraction of data is already supported. But with maxwell and any other similar service, this processor will be helpful. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1058890770


   @nsivabalan  please take another look when free :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data post process before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059983788


   hi @nsivabalan  any orther concern ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055746217


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057736428


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055741617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055739051


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056352114


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055985889


   @pratyakshsharma : Can you assist in reviewing this patch. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055739051


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057853853


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057734795


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057926126


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501",
       "triggerID" : "1057925985",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057981614


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501",
       "triggerID" : "1057925985",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pratyakshsharma commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059926596


   @wangxianghu The changes look good to me. I have a high level query. You have mentioned facing issues with binlog json format for example. Do you mean to say transformation is not possible with data in binlog json format? Data coming from binlogs also has a structure/schema assigned to it as far as I remember. Can you post a sample event where you feel this new PostProcessor you introduced might be useful? 
   Basically I want to understand the motivation behind introducing this PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu edited a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu edited a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059934053


   > @wangxianghu The changes look good to me. I have a high level query. You have mentioned facing issues with binlog json format for example. Do you mean to say transformation is not possible with data in binlog json format? Data coming from binlogs also has a structure/schema assigned to it as far as I remember. Can you post a sample event where you feel this new PostProcessor you introduced might be useful? Basically I want to understand the motivation behind introducing this PR.
   
   It is possible to deal with data in binlog json format, but not very convenient.
   
   1. For maxwell(our company use it to capture changed data)
   ```
   {
       "database": "test",
       "table": "maxwell",
       "type": "update",
       "ts": 1449786341,
       "xid": 940786,
       "commit": true,
       "data": {"id":1, "daemon": "Firebus!  Firebus!","update_time" : "2022-02-03 12:22:42"},
       "old":  {"daemon": "Stanislaw Lem"}
     }
   ```
   all we want is just :
   
   ```
   {
       "id": 1, 
       "daemon": "Firebus!  Firebus!", 
       "update_time": "2022-02-03 12:22:42"
   }
   ```
   we can add write a processor to extract the data from the entire json and maybe do some custom define process, without configuring a huge schema file(including all the fields in the binlog json, no matter if we need them or not)
   
   2. in some scenes, we need to encode some fileds for safety purpose, the processor can help us
   
   3. sometimes our data quality is not very well, some key field let's say precombine field have null value, we can use processor to fix it
   
   4. when our schema is read from jdbc or hive, we can adjust our kafka data compatible to it. 
   
   All in all, with custom processor we can do anything we want on the incoming json data before they are converted into DataSet
   Of course `Transformer` is a very useful feature too, but it is based on Spark DataSet, and have certain requirements for data quality.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056357002


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pratyakshsharma commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056358823


   > @pratyakshsharma : Can you assist in reviewing this patch.
   
   Ack. Will review it today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on a change in pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#discussion_r818363121



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -189,6 +189,11 @@ public static long totalNewMessages(OffsetRange[] ranges) {
             .defaultValue(KafkaResetOffsetStrategies.LATEST)
             .withDocumentation("Kafka consumer strategy for reading data.");
 
+    public static final ConfigProperty<String> JSON_KAFKA_PROCESSOR_CLASS_OPT = ConfigProperty
+        .key("hoodie.deltastreamer.source.kafka.json.processor.class")

Review comment:
       > hoodie.deltastreamer.source.json.kafka.post.processor.class
   
   done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055743969


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055746217


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu edited a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu edited a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1059934053


   > @wangxianghu The changes look good to me. I have a high level query. You have mentioned facing issues with binlog json format for example. Do you mean to say transformation is not possible with data in binlog json format? Data coming from binlogs also has a structure/schema assigned to it as far as I remember. Can you post a sample event where you feel this new PostProcessor you introduced might be useful? Basically I want to understand the motivation behind introducing this PR.
   
   It is possible to deal with data in binlog json format, but not very convenient.
   
   1. For maxwell(our company use it to capture changed data)
   ```
   {
       "database": "test",
       "table": "maxwell",
       "type": "update",
       "ts": 1449786341,
       "xid": 940786,
       "commit": true,
       "data": {"id":1, "daemon": "Firebus!  Firebus!","update_time" : "2022-02-03 12:22:42"},
       "old":  {"daemon": "Stanislaw Lem"}
     }
   ```
   all we want is just :
   
   ```
   {
       "id": 1, 
       "daemon": "Firebus!  Firebus!", 
       "update_time": "2022-02-03 12:22:42"
   }
   ```
   we can add write a processor to extract the data from the entire json and maybe do some custom define process, without configuring a huge schema file(including all the fields in the binlog json, no matter if we need them or not)
   
   2. in some scenes, we need to encode some fileds for safety purpose, the processor can help us
   
   3. sometimes our data quality is not very well, some key field let's say precombine field have null value, we can use processor to fix it
   
   4. when our schema is read from jdbc or hive, we can use processor adjust our kafka data compatible to it. 
   
   All in all, with custom processor we can do anything we want on the incoming json data before they are converted into DataSet
   Of course `Transformer` is a very useful feature too, but it is based on Spark DataSet, and have certain requirements for data quality.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055840859


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1056357002


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057790592


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#discussion_r818242568



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -189,6 +189,11 @@ public static long totalNewMessages(OffsetRange[] ranges) {
             .defaultValue(KafkaResetOffsetStrategies.LATEST)
             .withDocumentation("Kafka consumer strategy for reading data.");
 
+    public static final ConfigProperty<String> JSON_KAFKA_PROCESSOR_CLASS_OPT = ConfigProperty
+        .key("hoodie.deltastreamer.source.kafka.json.processor.class")
+        .noDefaultValue()
+        .withDocumentation("Json kafka source pre-processor class name, used to do pre-process on kafka data.");

Review comment:
       lets not use "pre-process" as it might be confusing. we can just say post process data after consuming from source and before giving it to deltastreamer. 

##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -189,6 +189,11 @@ public static long totalNewMessages(OffsetRange[] ranges) {
             .defaultValue(KafkaResetOffsetStrategies.LATEST)
             .withDocumentation("Kafka consumer strategy for reading data.");
 
+    public static final ConfigProperty<String> JSON_KAFKA_PROCESSOR_CLASS_OPT = ConfigProperty
+        .key("hoodie.deltastreamer.source.kafka.json.processor.class")

Review comment:
       hoodie.deltastreamer.source.json.kafka.post.processor.class




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data post process before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4930:
URL: https://github.com/apache/hudi/pull/4930


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057736428


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b31e57b6d3c2981e72c164f9844c82ff9b87ec68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455) 
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] wangxianghu commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057734099


   > Can we add a test case by adding a test JsonKafkaPostProcessor and ensure it works. also add a test where you set some invalid class for the new config added. and assert for exception.
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057790592


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055793248


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1057926126


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6455",
       "triggerID" : "b31e57b6d3c2981e72c164f9844c82ff9b87ec68",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489",
       "triggerID" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492",
       "triggerID" : "1057788742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501",
       "triggerID" : "1057925985",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 2f11c3d2cb94ca5c9f1a7f1144a80efb1ef6f063 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6489) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6492) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6501) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourcePostProcessor to support data preprocess before it is transformed to DataSet

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055840859


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6444) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055743969


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "902d714fb5f9cc1344603b460b29fe8d70bdb32a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   * 902d714fb5f9cc1344603b460b29fe8d70bdb32a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4930: [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data prepro…

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4930:
URL: https://github.com/apache/hudi/pull/4930#issuecomment-1055741617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443",
       "triggerID" : "cc69df077e45bde0ce2f3789d6b59dc53eb0825f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc69df077e45bde0ce2f3789d6b59dc53eb0825f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6443) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org