You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bajiaolong (via GitHub)" <gi...@apache.org> on 2023/11/01 03:02:52 UTC
[I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
bajiaolong opened a new issue, #9965:
URL: https://github.com/apache/hudi/issues/9965
At present, I am sending the data of the whole database to a topic of kafka. I want to use flink to read kafka data and write the data into multiple tables of hudi. Do you have any idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1894912026
@bajiaolong Closing out this, Please reopen or create a new one for further queries. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1789113863
@bajiaolong One way would be filter the stream by table name and then ingest to hudi table. @danny0405 Any other way to achieve this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1803084529
yeah, already on the list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "bajiaolong (via GitHub)" <gi...@apache.org>.
bajiaolong commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1801269277
@ad1happy2go
Is it possible to support multi-table synchronization in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "bajiaolong (via GitHub)" <gi...@apache.org>.
bajiaolong commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1789890659
> How many tables are there in your database, it is feasible if you have just handful of tables like 20, then you can consume the Kafka topic and partition the stream by table name, for each partitioned stream, you can pipeline with hudi sink, you need to write some DataStream pipelines manually, take https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/HoodiePipeline.java for a single table example.
1. Why is the data in the table limited to 20 and what is the reason.
2. Now the data of all tables in my library are synchronized to one table. Partition is done with the schema and table name of the database. When reading downstream, I filter the table name through stream. However, this method is very time-consuming. Is there a stream read operation that only reads fixed partitions, so that I can get a single table
3. Do you have any suggestions for the second one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1801186970
@bajiaolong
1. Its not limited to 20, I guess what @danny0405 meant is number of tables should be handful as you may need to manage those number of streams.
2. Not sure if there is a way to only read specific partitions from kafka topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #9965: [SUPPORT] Solution for synchronizing the entire database table in flink
URL: https://github.com/apache/hudi/issues/9965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9965:
URL: https://github.com/apache/hudi/issues/9965#issuecomment-1789875033
How many tables are there in your database, it is feasible if you have just handful of tables like 20, then you can consume the Kafka topic and partition the stream by table name, for each partitioned stream, you can pipeline with hudi sink, you need to write some DataStream pipelines manually, take https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/HoodiePipeline.java for a single table example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org