You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "complone (via GitHub)" <gi...@apache.org> on 2023/04/17 09:18:08 UTC
[GitHub] [flink-connector-jdbc] complone opened a new pull request, #42: [FLINK-31820] Support data source sub-database and sub-table
complone opened a new pull request, #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42
related to [FLINK-31820](https://issues.apache.org/jira/browse/FLINK-31820)
#### Background & Motivation
Hi, community. As I develop full-database synchronization syntax in the company to reduce the increase in the number of connections when multiple data sources are synchronized, I find that traditional databases (Mysql, Postgres, Oracle) need to support the function of sub-database and sub-table. It supports data source sub-database sub-table on top of existing projects
#### Goals
- Support data source sub-database sub-table
- When the sub-database and table functions are realized, the original data fragments allocated according to the partition are guaranteed to be available
- Prioritize to ensure that Mysql, Postgres, Oracle support sub-database sub-table
#### Change
- Every time JdbcRowDataInputFormat switches fragments, it will call open(InputSplit inputSplit) once (corresponding to closing the current data fragmentation method: close()), the value of inputSplit corresponds to the value of x in Serializable[x][y] is incremented, and each parallel The instance will not be executed repeatedly. For example, if there are 1024 split tables and each table has 2 data splits, then the value range of inputSplit.getSplitNumber() is: [0~2047]. Therefore, when the database is divided into tables, it should be ensured that the query logic of each table after the database and table is consistent with the query logic of the original single database and single table.
- JdbcMultiTableProvider needs to split the Serializable[][] twice based on the results of the original data fragmentation, according to the sub-database and sub-table, to ensure that JdbcRowDataInputFormat can obtain the split table information
#### Design
According to the configured url and table_name expressions, the basic encoding steps are as follows:
- Query all schemas in the database
- Match the schema by regular
- Query matches the table under the schema
- By regular matching table
- Return the corresponding relationship between the database url and the table: List<TableItem>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] MartijnVisser commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1592942865
No consensus yet in ticket, closing PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] boring-cyborg[bot] commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1510993607
Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] complone commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "complone (via GitHub)" <gi...@apache.org>.
complone commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1511038266
> @complone请按照https://flink.apache.org/how-to-contribute/overview/进行任何代码贡献
I have explained the changes according to the community contribution manual. Currently, the single test has not been fully completed. Please wait for a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] MartijnVisser closed pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser closed pull request #42: [FLINK-31820] Support data source sub-database and sub-table
URL: https://github.com/apache/flink-connector-jdbc/pull/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] MartijnVisser commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1511509388
> I have explained the changes according to the community contribution manual.
Please check the commit naming convention. Also, see my remark in the original Jira ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] MartijnVisser commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1511000963
@complone Please follow https://flink.apache.org/how-to-contribute/overview/ for any code contribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-jdbc] complone commented on pull request #42: [FLINK-31820] Support data source sub-database and sub-table
Posted by "complone (via GitHub)" <gi...@apache.org>.
complone commented on PR #42:
URL: https://github.com/apache/flink-connector-jdbc/pull/42#issuecomment-1518912732
> 根据社区贡献手册解释了变化
> community contribution manual
Thank you, please let me apologize for not understanding the community contribution manual. Since I need to develop more functions for my company recently, I haven't had time to read your content in JIRA, I will read the manual recently and explain clearly Related concepts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org