You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/09/06 15:36:57 UTC
[GitHub] [hudi] pratyakshsharma opened a new pull request #2073: [HUDI-769]: added blog for HoodieMultiTableDeltaStreamer
pratyakshsharma opened a new pull request #2073:
URL: https://github.com/apache/hudi/pull/2073
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490166850
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
+
+This blog will guide you through configuring and running `HoodieMultiTableDeltaStreamer`.
+
+### Configuration
+
+ - `HoodieMultiTableDeltaStreamer` expects users to maintain table wise overridden properties in separate files in a dedicated config folder. Common properties can be configured via common properties file also.
+ - By default, hudi datasets are created under the path `<base-path-prefix>/<database_name>/<name_of_table_to_be_ingested>`.
+ - You need to provide the names of tables to be ingested via the property `hoodie.deltastreamer.ingestion.tablesToBeIngested` in the format `<database>.<table>`, for example
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db2.table2
+```
+
+ - If you do not provide database name, then it is assumed the table belongs to default database and the hudi dataset for the concerned table is created under the path `<base-path-prefix>/default/<name_of_table_to_be_ingested>`. Also there is a provision to override the default path for hudi datasets. You can create hudi dataset for a particular table by setting the property `hoodie.deltastreamer.ingestion.targetBasePath` in table level config file
+ - There are a lot of properties that one might like to override per table, for example
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic2
+hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
+hoodie.deltastreamer.keygen.timebased.input.dateformat=yyyy-MM-dd HH:mm:ss.S
+hoodie.datasource.hive_sync.table=short_trip_uber_hive_dummy_table
+hoodie.deltastreamer.ingestion.targetBasePath=s3:///temp/hudi/table1
+```
+
+ - Properties like above need to be set for every table to be ingested. As already suggested at the beginning, users are expected to maintain separate config files for every table by setting the below property
+
+```java
+hoodie.deltastreamer.ingestion.<db>.<table>.configFile=s3:///tmp/config/config1.properties
+```
+
+If you do not want to set the above property for every table, you can simply create config files for every table to be ingested under the config folder with the name - `<database>_<table>_config.properties`. For example if you want to ingest table1 and table2 from dummy database, where config folder is set to `s3:///tmp/config`, then you need to create 2 config files on the given paths - `s3:///tmp/config/dummy_table1_config.properties` and `s3:///tmp/config/dummy_table2_config.properties`.
+
+ - Finally you can specify all the common properties in a common properties file. Common properties file does not necessarily have to lie under config folder but it is advised to keep it along with other config files. This file will contain the below properties
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db2.table2
+hoodie.deltastreamer.ingestion.db1.table1.configFile=s3:///tmp/config_table1.properties
+hoodie.deltastreamer.ingestion.db2.table2.configFile=s3:///tmp/config_table2.properties
+```
+
+### Run Command
+
+`HoodieMultiTableDeltaStreamer` can be run similar to how one runs `HoodieDeltaStreamer`. Please refer to the example given below for the command.
+
+
+### Example
+
+Suppose you want to ingest table1 and table2 from db1 and want to ingest the 2 tables under the path `s3:///temp/hudi`. You can ingest them using the below command
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props s3:///temp/hudi-ingestion-config/kafka-source.properties \
+ --config-folder s3:///temp/hudi-ingestion-config \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --base-path-prefix s3:///temp/hudi \
+ --target-table dummy_table \
+ --op UPSERT
+```
+
+s3:///temp/config/kafka-source.properties
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db1.table2
+hoodie.deltastreamer.ingestion.db1.table1.configFile=s3:///temp/hudi-ingestion-config/config_table1.properties
+hoodie.deltastreamer.ingestion.db21.table2.configFile=s3:///temp/hudi-ingestion-config/config_table2.properties
+
+#Kafka props
+bootstrap.servers=localhost:9092
+auto.offset.reset=earliest
+schema.registry.url=http://localhost:8081
+
+hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
+```
+
+s3:///temp/hudi-ingestion-config/config_table1.properties
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key1
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic1
+```
+
+s3:///temp/hudi-ingestion-config/config_table2.properties
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key2
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic2
+```
Review comment:
Ok, makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490165732
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
+
+This blog will guide you through configuring and running `HoodieMultiTableDeltaStreamer`.
+
+### Configuration
+
+ - `HoodieMultiTableDeltaStreamer` expects users to maintain table wise overridden properties in separate files in a dedicated config folder. Common properties can be configured via common properties file also.
+ - By default, hudi datasets are created under the path `<base-path-prefix>/<database_name>/<name_of_table_to_be_ingested>`.
Review comment:
Doing that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2073: [HUDI-769]: added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r484584835
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
Review comment:
`Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.` Can we use some bold keywords?
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
+
+This blog will guide you through configuring and running `HoodieMultiTableDeltaStreamer`.
+
+### Configuration
+
+ - `HoodieMultiTableDeltaStreamer` expects users to maintain table wise overridden properties in separate files in a dedicated config folder. Common properties can be configured via common properties file also.
+ - By default, hudi datasets are created under the path `<base-path-prefix>/<database_name>/<name_of_table_to_be_ingested>`.
Review comment:
IMO, this item should not be a single one, right? It's an explanation, not about a configuration. It would be better to merge it with the third one. WDYT?
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
+
+This blog will guide you through configuring and running `HoodieMultiTableDeltaStreamer`.
+
+### Configuration
+
+ - `HoodieMultiTableDeltaStreamer` expects users to maintain table wise overridden properties in separate files in a dedicated config folder. Common properties can be configured via common properties file also.
+ - By default, hudi datasets are created under the path `<base-path-prefix>/<database_name>/<name_of_table_to_be_ingested>`.
+ - You need to provide the names of tables to be ingested via the property `hoodie.deltastreamer.ingestion.tablesToBeIngested` in the format `<database>.<table>`, for example
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db2.table2
+```
+
+ - If you do not provide database name, then it is assumed the table belongs to default database and the hudi dataset for the concerned table is created under the path `<base-path-prefix>/default/<name_of_table_to_be_ingested>`. Also there is a provision to override the default path for hudi datasets. You can create hudi dataset for a particular table by setting the property `hoodie.deltastreamer.ingestion.targetBasePath` in table level config file
+ - There are a lot of properties that one might like to override per table, for example
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic2
+hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
+hoodie.deltastreamer.keygen.timebased.input.dateformat=yyyy-MM-dd HH:mm:ss.S
+hoodie.datasource.hive_sync.table=short_trip_uber_hive_dummy_table
+hoodie.deltastreamer.ingestion.targetBasePath=s3:///temp/hudi/table1
+```
+
+ - Properties like above need to be set for every table to be ingested. As already suggested at the beginning, users are expected to maintain separate config files for every table by setting the below property
+
+```java
+hoodie.deltastreamer.ingestion.<db>.<table>.configFile=s3:///tmp/config/config1.properties
+```
+
+If you do not want to set the above property for every table, you can simply create config files for every table to be ingested under the config folder with the name - `<database>_<table>_config.properties`. For example if you want to ingest table1 and table2 from dummy database, where config folder is set to `s3:///tmp/config`, then you need to create 2 config files on the given paths - `s3:///tmp/config/dummy_table1_config.properties` and `s3:///tmp/config/dummy_table2_config.properties`.
+
+ - Finally you can specify all the common properties in a common properties file. Common properties file does not necessarily have to lie under config folder but it is advised to keep it along with other config files. This file will contain the below properties
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db2.table2
+hoodie.deltastreamer.ingestion.db1.table1.configFile=s3:///tmp/config_table1.properties
+hoodie.deltastreamer.ingestion.db2.table2.configFile=s3:///tmp/config_table2.properties
+```
+
+### Run Command
+
+`HoodieMultiTableDeltaStreamer` can be run similar to how one runs `HoodieDeltaStreamer`. Please refer to the example given below for the command.
+
+
+### Example
+
+Suppose you want to ingest table1 and table2 from db1 and want to ingest the 2 tables under the path `s3:///temp/hudi`. You can ingest them using the below command
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props s3:///temp/hudi-ingestion-config/kafka-source.properties \
+ --config-folder s3:///temp/hudi-ingestion-config \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --base-path-prefix s3:///temp/hudi \
+ --target-table dummy_table \
+ --op UPSERT
+```
+
+s3:///temp/config/kafka-source.properties
+
+```java
+hoodie.deltastreamer.ingestion.tablesToBeIngested=db1.table1,db1.table2
+hoodie.deltastreamer.ingestion.db1.table1.configFile=s3:///temp/hudi-ingestion-config/config_table1.properties
+hoodie.deltastreamer.ingestion.db21.table2.configFile=s3:///temp/hudi-ingestion-config/config_table2.properties
+
+#Kafka props
+bootstrap.servers=localhost:9092
+auto.offset.reset=earliest
+schema.registry.url=http://localhost:8081
+
+hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
+```
+
+s3:///temp/hudi-ingestion-config/config_table1.properties
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key1
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic1
+```
+
+s3:///temp/hudi-ingestion-config/config_table2.properties
+
+```java
+hoodie.datasource.write.recordkey.field=_row_key2
+hoodie.datasource.write.partitionpath.field=created_at
+hoodie.deltastreamer.source.kafka.topic=topic2
+```
Review comment:
Maybe add more information looks better? e.g. the development of this feature in the future? And not end with code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490670265
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,104 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyaksh.sharma.hudi
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports **COPY_ON_WRITE** storage type only and the ingestion is done in a **sequential** way.
Review comment:
my 2 cents. take your call though. Instead of "one of the most common problems that one faces is simplifying the onboarding process for multiple tables", how about "one of the major challenges is the complexity in onboarding multiple tables".
##########
File path: content/blog/ingest-multiple-tables-using-hudi/index.html
##########
@@ -0,0 +1,233 @@
+<!doctype html>
Review comment:
yeah, I don't think you need this file.
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under `hud
--op BULK_INSERT
```
+For detailed information on how to configure and use `HoodieMultiTableDeltaStreamer`, please refer [blog section](/blog/ingest-multiple-tables-using-hudi).
Review comment:
I don't know, just confirming. do you know this link actually works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490660995
##########
File path: content/blog/ingest-multiple-tables-using-hudi/index.html
##########
@@ -0,0 +1,233 @@
+<!doctype html>
Review comment:
Hi, It seems we do not need the *.html file. You can refer to this PR #1992
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490165008
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,101 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyakshsharma
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports COPY_ON_WRITE storage type only and the ingestion is done in a sequential way.
Review comment:
Sure. Thank you for pointing this out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #2073: [HUDI-769] Add blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #2073:
URL: https://github.com/apache/hudi/pull/2073
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#issuecomment-694335686
@yanghua @bhasudha Please take a pass. I am not sure if I added the index.html in proper form or not. Please suggest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490163062
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under `hud
--op BULK_INSERT
```
+For detailed information on how to configure and use `HoodieMultiTableDeltaStreamer`, please refer blog section.
Review comment:
Looks like I need to generate index.html file to be able to add the link. @nsivabalan Can you please guide me here? Do I need to write the html manually or is there some alternate way of generating the html for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#issuecomment-694834432
@yanghua Please take a pass. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490163062
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under `hud
--op BULK_INSERT
```
+For detailed information on how to configure and use `HoodieMultiTableDeltaStreamer`, please refer blog section.
Review comment:
Looks like I need to generate index.html file to be able to add the link. @nsivabalan Can you please guide me here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490908891
##########
File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md
##########
@@ -0,0 +1,104 @@
+---
+title: "Ingest multiple tables using Hudi"
+excerpt: "Ingesting multiple tables using Hudi at a single go is now possible. This blog gives a detailed explanation of how to achieve the same using `HoodieMultiTableDeltaStreamer.java`"
+author: pratyaksh.sharma.hudi
+category: blog
+---
+
+When building a change data capture pipeline for already existing or newly created relational databases, one of the most common problems that one faces is simplifying the onboarding process for multiple tables. Ingesting multiple tables to Hudi dataset at a single go is now possible using `HoodieMultiTableDeltaStreamer` class which is a wrapper on top of the more popular `HoodieDeltaStreamer` class. Currently `HoodieMultiTableDeltaStreamer` supports **COPY_ON_WRITE** storage type only and the ingestion is done in a **sequential** way.
Review comment:
Guess will keep it in its original form. :) Thank you for your kind suggestion though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490908302
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under `hud
--op BULK_INSERT
```
+For detailed information on how to configure and use `HoodieMultiTableDeltaStreamer`, please refer [blog section](/blog/ingest-multiple-tables-using-hudi).
Review comment:
yeah, it works :) I have tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2073: [HUDI-769]: added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r484584030
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under `hud
--op BULK_INSERT
```
+For detailed information on how to configure and use `HoodieMultiTableDeltaStreamer`, please refer blog section.
Review comment:
It would be better to add a link to the address of the blog. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #2073:
URL: https://github.com/apache/hudi/pull/2073#discussion_r490908076
##########
File path: content/blog/ingest-multiple-tables-using-hudi/index.html
##########
@@ -0,0 +1,233 @@
+<!doctype html>
Review comment:
Thank you for pointing the PR. Removed the file now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org