You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/31 06:58:51 UTC

[GitHub] [hudi] kywe665 opened a new pull request #4481: [HUDI-3075] [WIP] - Docs for Debezium DeltaStreamer Source

kywe665 opened a new pull request #4481:
URL: https://github.com/apache/hudi/pull/4481


   ## What is the purpose of the pull request
   
   docs for debezium source
   
   ## Brief change log
   
   added docs for debezium source
   
   ## Verify this pull request
   
   docs changes only
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #4481: [HUDI-3075] [WIP] - Docs for Debezium DeltaStreamer Source

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4481:
URL: https://github.com/apache/hudi/pull/4481#discussion_r779915089



##########
File path: website/docs/hoodie_deltastreamer.md
##########
@@ -336,6 +336,29 @@ jobs: `hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'`
 Spark SQL should be configured using this hoodie config:
 hoodie.deltastreamer.source.sql.sql.query = 'select * from source_table'
 
+### Debezium Sources
+Debezium is an open source distributed platform for change data capture(CDC). Hudi has both a PostgresDebeziumSource and a 

Review comment:
       can we specify the full class name for these sources? 

##########
File path: website/docs/hoodie_deltastreamer.md
##########
@@ -336,6 +336,29 @@ jobs: `hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'`
 Spark SQL should be configured using this hoodie config:
 hoodie.deltastreamer.source.sql.sql.query = 'select * from source_table'
 
+### Debezium Sources
+Debezium is an open source distributed platform for change data capture(CDC). Hudi has both a PostgresDebeziumSource and a 
+MysqlDebeziumSource. With these sources, we can continuously capture row-level changes that insert, update and delete 
+records that were committed to a MySQL or Postgres database and seamlessly apply these changes to Hudi tables.
+
+[Debezium](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) is implemented as a Kafka 
+connect source, that reads change logs from databases ([logical decoding](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)
+in PostgreSQL and `binlog` in MySQL) and ingests them into a kafka topic. Debezium uses a single kafka topic per table in the source database.
+
+![debezium](/assets/images/debezium_arch.png)
+
+The connector generates data change event records and streams them to Kafka topics. For each table, the default behavior 
+is that the connector streams all generated events to a separate Kafka topic for that table. In addition, Debezium 
+registers the schema of the change events in kafka to a schema registry, such as Confluent schema registry.
+
+#### Configuration 
+#### --@Rajesh, we need to make this more specific to the code and put some sample code 

Review comment:
       we probably need to clean this up more?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] kywe665 commented on pull request #4481: [HUDI-3075] [WIP] - Docs for Debezium DeltaStreamer Source

Posted by GitBox <gi...@apache.org>.
kywe665 commented on pull request #4481:
URL: https://github.com/apache/hudi/pull/4481#issuecomment-1003289656


   @rmahindra123 see this WIP for Debezium source docs. I tagged a spot where I need help generating sample code and accurately documenting the parameters the user has to configure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on pull request #4481: [HUDI-3075] [WIP] - Docs for Debezium DeltaStreamer Source

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4481:
URL: https://github.com/apache/hudi/pull/4481#issuecomment-1041107793


   @kywe665 : Can we take this forward. guess its been there for sometime. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org