You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/06/18 21:27:02 UTC

[GitHub] [hudi] sbernauer opened a new pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection

sbernauer opened a new pull request #3111:
URL: https://github.com/apache/hudi/pull/3111


   Hi Hudi Team!
   
   ## What is the purpose of the pull request
   We are using Deltastreamer to ingest events from kafka into an S3 store.
   We had some incident this week after some of our producers switched to a new schema version, but some remained on older schem versions.
   
   We stumbled over this PR https://github.com/apache/hudi/pull/2619. With the following settings we can enbale the Custom avro kafka deserializer
   ```
   hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer
   hoodie.deltastreamer.schemaprovider.class=org.apache.hudi.utilities.schema.FilebasedSchemaProvider
   ```
   
   Doing so we noticed that KafkaAvroSchemaDeserializer tries instanciating the configured SchemaProvider via reflection. It calls the constructor in https://github.com/apache/hudi/blob/cdb9b48170ef98634babd8954392efb1c1b90fcf/hudi-utilities/src/main/java/org/apache/hudi/utilities/deser/KafkaAvroSchemaDeserializer.java#L55 resulting in
   ```
   Caused by: java.lang.NoSuchMethodException: org.apache.hudi.utilities.schema.SchemaProviderWithPostProcessor.<init>(org.apache.hudi.common.config.TypedProperties)
   ```
   There are multiple problems here
   * The Class has not the needed constructor
   * As we are using FilebasedSchemaProvider it calls the wrong class wrapping the FilebasedSchemaProvider
   * The FilebasedSchemaProvider needs the JavaSparkContext to work correctly. I think we dont have access to the JavaSparkContext in KafkaAvroSchemaDeserializer
   
   ## Brief change log
   
   This PR modifies the KafkaAvroSchemaDeserializer, so that it does not rely on Reflection to call the SchemaProvider.
   Instead it uses the normal progam flow to ask the SchemProvider for the sourceSchema.
   It then passes the sourceSchema as Property to the KafkaAvroSchemaDeserializer so that it can be used for deserialization.
   
   Anyway: I wonder if it is a good idea to make the usage of KafkaAvroSchemaDeserializer the default. IMHO it woud make sense.
   
   ## Verify this pull request
   You have to enable the feature with
   ```
   hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer
   ```
   
   The tests are WIP
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-867622402


   We always have a practice of creating a jira for every patch. especially bugs. I have created one and prefixed the title. Will fix the commit msg as well. Something to remember for future :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864362419


   @hudi-bot run travis


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-867560326


   So, the tests passed with the third attempt :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658273889



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       okay makes sense. Having this with options is kind of confusing and I thought this is user configurable. 
   
   While I have both of you here, are we turning this on by default now?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3111](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (38470b5) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `17.03%`.
   > The diff coverage is `40.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3111/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #3111       +/-   ##
   =============================================
   - Coverage     45.85%   28.81%   -17.04%     
   + Complexity     5269     1254     -4015     
   =============================================
     Files           908      370      -538     
     Lines         39332    14099    -25233     
     Branches       4239     1441     -2798     
   =============================================
   - Hits          18036     4063    -13973     
   + Misses        19451     9739     -9712     
   + Partials       1845      297     -1548     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hudicli | `?` | |
   | hudiclient | `22.29% <ø> (-8.15%)` | :arrow_down: |
   | hudicommon | `?` | |
   | hudiflink | `?` | |
   | hudihadoopmr | `?` | |
   | hudisparkdatasource | `?` | |
   | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
   | huditimelineservice | `?` | |
   | hudiutilities | `58.25% <40.00%> (+1.61%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `86.48% <50.00%> (-0.79%)` | :arrow_down: |
   | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `81.25% <100.00%> (-2.09%)` | :arrow_down: |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [612 more](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cdb9b48...38470b5](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3111](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (38470b5) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `42.71%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3111/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #3111       +/-   ##
   ============================================
   - Coverage     45.85%   3.13%   -42.72%     
   + Complexity     5269      82     -5187     
   ============================================
     Files           908     274      -634     
     Lines         39332   10648    -28684     
     Branches       4239    1089     -3150     
   ============================================
   - Hits          18036     334    -17702     
   + Misses        19451   10288     -9163     
   + Partials       1845      26     -1819     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hudicli | `?` | |
   | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: |
   | hudicommon | `?` | |
   | hudiflink | `?` | |
   | hudihadoopmr | `?` | |
   | hudisparkdatasource | `?` | |
   | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
   | huditimelineservice | `?` | |
   | hudiutilities | `9.52% <0.00%> (-47.12%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
   | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [749 more](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cdb9b48...38470b5](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-866598757


   I already had checked, it timed out.
   I also asked the Hudi bot to rerun the test but they didn't.
   I think I will have to amend the last commit and force push to trigger the tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r659691658



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       Created https://github.com/apache/hudi/pull/3172 that adresses this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38470b55ab1b848430425665d99f394a56ef619c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=401",
       "triggerID" : "38470b55ab1b848430425665d99f394a56ef619c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38470b55ab1b848430425665d99f394a56ef619c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=401) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658274376



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -176,7 +176,9 @@ public KafkaOffsetGen(TypedProperties props) {
     props.keySet().stream().filter(prop -> {
       // In order to prevent printing unnecessary warn logs, here filter out the hoodie
       // configuration items before passing to kafkaParams
-      return !prop.toString().startsWith("hoodie.");
+      return !prop.toString().startsWith("hoodie.")
+              // We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
+              || prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");

Review comment:
       understood. my point was to pull these into static final string constants. So its easy to trace if more places start using them




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38470b55ab1b848430425665d99f394a56ef619c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "38470b55ab1b848430425665d99f394a56ef619c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b47c4d4f0bdbfbd7ed0f8038a9e145192632674 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372) 
   * 38470b55ab1b848430425665d99f394a56ef619c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-866599382


   @hudi-bot run travis


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658281970



##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/deser/TestKafkaAvroSchemaDeserializer.java
##########
@@ -102,9 +99,10 @@ private IndexedRecord createExtendUserRecord() {
   public void testKafkaAvroSchemaDeserializer() {
     byte[] bytesOrigRecord;
     IndexedRecord avroRecord = createUserRecord();
-    SchemaTestProvider.schemaToReturn.set(origSchema);
-    KafkaAvroSchemaDeserializer avroDeserializer = new KafkaAvroSchemaDeserializer(schemaRegistry, new HashMap(defaultConfig));
-    avroDeserializer.configure(new HashMap(defaultConfig), false);
+    config.put("hoodie.deltastreamer.source.kafka.value.deserializer.schema", origSchema.toString());

Review comment:
       Looking at this as I rebase #2833 against this PR. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vburenin commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vburenin commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-867072324


   Test are definitely not ideal, I had a lot of failures around different places when the change was just to reduce info to debug level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vburenin commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vburenin commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864408558


   @sbernauer This is definitely a good change in my opinion. I like it. 
   
   Since you are using same approach to ingest data as we do, you might be interested to talk more about it, especially when it comes to the schema evolution. I have a bunch of changes on our end designed specifically for that, I bet it would be mutually beneficial to share them with you and later on upstream them. I sent you a ping e-mail to your email address you specified on github.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-867069605


   The previous test run failed because of the hudi-utilities. The current testrun failed two other tests, the hudi-utilities were fine. How do we progress here? I think the tests itself are problematic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658090982



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       Both settings (old and new) are not set via config. Both are calculated and (over)written by Deltastreamer itself. They are used to pass things around.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658063310



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       We did not replace A with B. we have a config that determines the schemaProvider class. unless someone sets this config value, behavior stays the same (old behavior).
   https://github.com/apache/hudi/blob/b32855545bb9dbc6f49c18d1586585a39abe5004/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java#L60
   check line 60 to 64. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38470b55ab1b848430425665d99f394a56ef619c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=401",
       "triggerID" : "38470b55ab1b848430425665d99f394a56ef619c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b47c4d4f0bdbfbd7ed0f8038a9e145192632674 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372) 
   * 38470b55ab1b848430425665d99f394a56ef619c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=401) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   * 8b47c4d4f0bdbfbd7ed0f8038a9e145192632674 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658093617



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -176,7 +176,9 @@ public KafkaOffsetGen(TypedProperties props) {
     props.keySet().stream().filter(prop -> {
       // In order to prevent printing unnecessary warn logs, here filter out the hoodie
       // configuration items before passing to kafkaParams
-      return !prop.toString().startsWith("hoodie.");
+      return !prop.toString().startsWith("hoodie.")
+              // We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
+              || prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");

Review comment:
       This expression must currently match two settings:
   `hoodie.deltastreamer.source.kafka.value.deserializer.class`
   and
   `hoodie.deltastreamer.source.kafka.value.deserializer.schema`
   So we ether list all the settings (may be more than two in the future) which is error-prone or we use the prefix here




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r659587977



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -176,7 +176,9 @@ public KafkaOffsetGen(TypedProperties props) {
     props.keySet().stream().filter(prop -> {
       // In order to prevent printing unnecessary warn logs, here filter out the hoodie
       // configuration items before passing to kafkaParams
-      return !prop.toString().startsWith("hoodie.");
+      return !prop.toString().startsWith("hoodie.")
+              // We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
+              || prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");

Review comment:
       Should i create a PR that pulls `hoodie.deltastreamer.source.kafka.value.deserializer.` to a static final string constant inside KafkaOffsetGen?
   That would be no problem




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b47c4d4f0bdbfbd7ed0f8038a9e145192632674 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658281481



##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/deser/TestKafkaAvroSchemaDeserializer.java
##########
@@ -102,9 +99,10 @@ private IndexedRecord createExtendUserRecord() {
   public void testKafkaAvroSchemaDeserializer() {
     byte[] bytesOrigRecord;
     IndexedRecord avroRecord = createUserRecord();
-    SchemaTestProvider.schemaToReturn.set(origSchema);
-    KafkaAvroSchemaDeserializer avroDeserializer = new KafkaAvroSchemaDeserializer(schemaRegistry, new HashMap(defaultConfig));
-    avroDeserializer.configure(new HashMap(defaultConfig), false);
+    config.put("hoodie.deltastreamer.source.kafka.value.deserializer.schema", origSchema.toString());

Review comment:
       I ll replace this with`DataSourceWriteOptions.KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA()` 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-867681473


   Many thanks @nsivabalan!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-866347883


   Yes, we tested it on our staging env for multiple streaming jobs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658034007



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       @nsivabalan do we need a story for deprecating this first? seems like we just replaced A with B and it means existing jobs are going to break out-of-box? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

vinothchandar commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658037497



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -176,7 +176,9 @@ public KafkaOffsetGen(TypedProperties props) {
     props.keySet().stream().filter(prop -> {
       // In order to prevent printing unnecessary warn logs, here filter out the hoodie
       // configuration items before passing to kafkaParams
-      return !prop.toString().startsWith("hoodie.");
+      return !prop.toString().startsWith("hoodie.")
+              // We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
+              || prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");

Review comment:
       reuse the property name from the other class, without hardcoding?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

nsivabalan merged pull request #3111:
URL: https://github.com/apache/hudi/pull/3111


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864363169


   CC @vburenin @nsivabalan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3111](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8b47c4d) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `29.35%`.
   > The diff coverage is `40.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3111/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #3111       +/-   ##
   =============================================
   - Coverage     45.85%   16.50%   -29.36%     
   + Complexity     5269      467     -4802     
   =============================================
     Files           908      274      -634     
     Lines         39332    10648    -28684     
     Branches       4239     1089     -3150     
   =============================================
   - Hits          18036     1757    -16279     
   + Misses        19451     8732    -10719     
   + Partials       1845      159     -1686     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hudicli | `?` | |
   | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: |
   | hudicommon | `?` | |
   | hudiflink | `?` | |
   | hudihadoopmr | `?` | |
   | hudisparkdatasource | `?` | |
   | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
   | huditimelineservice | `?` | |
   | hudiutilities | `58.29% <40.00%> (+1.65%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `86.48% <50.00%> (-0.79%)` | :arrow_down: |
   | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `81.25% <100.00%> (-2.09%)` | :arrow_down: |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [708 more](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cdb9b48...8b47c4d](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-commenter commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

codecov-commenter commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3111](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b96d84e) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `26.95%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3111/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #3111       +/-   ##
   =============================================
   - Coverage     45.85%   18.90%   -26.96%     
   + Complexity     5269      869     -4400     
   =============================================
     Files           908      368      -540     
     Lines         39332    13955    -25377     
     Branches       4239     1426     -2813     
   =============================================
   - Hits          18036     2638    -15398     
   + Misses        19451    11155     -8296     
   + Partials       1845      162     -1683     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hudicli | `?` | |
   | hudiclient | `22.27% <ø> (-8.18%)` | :arrow_down: |
   | hudicommon | `?` | |
   | hudiflink | `?` | |
   | hudihadoopmr | `?` | |
   | hudisparkdatasource | `?` | |
   | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
   | huditimelineservice | `?` | |
   | hudiutilities | `10.00% <0.00%> (-46.64%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
   | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [649 more](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cdb9b48...b96d84e](https://codecov.io/gh/apache/hudi/pull/3111?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r659589194



##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/deser/TestKafkaAvroSchemaDeserializer.java
##########
@@ -102,9 +99,10 @@ private IndexedRecord createExtendUserRecord() {
   public void testKafkaAvroSchemaDeserializer() {
     byte[] bytesOrigRecord;
     IndexedRecord avroRecord = createUserRecord();
-    SchemaTestProvider.schemaToReturn.set(origSchema);
-    KafkaAvroSchemaDeserializer avroDeserializer = new KafkaAvroSchemaDeserializer(schemaRegistry, new HashMap(defaultConfig));
-    avroDeserializer.configure(new HashMap(defaultConfig), false);
+    config.put("hoodie.deltastreamer.source.kafka.value.deserializer.schema", origSchema.toString());

Review comment:
       Alright thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r659591466



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       I'm fine to define the config name in an other place (e.g. in KafkaOffsetGen). Should I change it with a new PR?
   I would suggest turning it on by default, what are your thoughts @nsivabalan and @vburenin?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r659691814



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -176,7 +176,9 @@ public KafkaOffsetGen(TypedProperties props) {
     props.keySet().stream().filter(prop -> {
       // In order to prevent printing unnecessary warn logs, here filter out the hoodie
       // configuration items before passing to kafkaParams
-      return !prop.toString().startsWith("hoodie.");
+      return !prop.toString().startsWith("hoodie.")
+              // We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
+              || prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");

Review comment:
       Created https://github.com/apache/hudi/pull/3172 that addresses this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-866384611


   awesome 👍 Can you please check why CI is failing. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot edited a comment on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=271",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "864362419",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b96d84eae7cc58b005f00986786c6865f7909a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272",
       "triggerID" : "866599382",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372",
       "triggerID" : "8b47c4d4f0bdbfbd7ed0f8038a9e145192632674",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b96d84eae7cc58b005f00986786c6865f7909a2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=272) 
   * 8b47c4d4f0bdbfbd7ed0f8038a9e145192632674 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=372) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] sbernauer commented on a change in pull request #3111: [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

sbernauer commented on a change in pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#discussion_r658090982



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -382,8 +382,6 @@ object DataSourceWriteOptions {
 
   // Avro Kafka Source configs
   val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-
-  // Schema provider class to be set to be used in custom kakfa deserializer
-  val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"

Review comment:
       Both settings (old and new) are not set via config. Bot are calculated and (over)written by Deltastreamer itself. They are used to pass things around.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection

Posted by GitBox <gi...@apache.org>.

hudi-bot commented on pull request #3111:
URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8985d48eb28ad448a6ec6607fd20c3efcc92b3d2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8985d48eb28ad448a6ec6607fd20c3efcc92b3d2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org