You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "WinkerDu (via GitHub)" <gi...@apache.org> on 2023/04/19 15:03:05 UTC
[GitHub] [doris] WinkerDu opened a new pull request, #18834: Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
WinkerDu opened a new pull request, #18834:
URL: https://github.com/apache/doris/pull/18834
# Proposed changes
Issue Number: close #xxx
## Problem summary
In my organization scene, We need to specify databases that are excluded to synchronize to doris, like some databases store temporary table.
Since #17803 introduce `specified_database_list` to specify 'include databases', this pr introduce new config `exclude_database_list` to specify 'exclude databases', and rename `specified_database_list` to `include_database_list` for naming symmetry.
BTW, when `include_database_list` and `exclude_database_list` specify overlapping databases, `exclude_database_list` would take effect with higher privilege over `include_database_list`.
## Checklist(Required)
* [x] Does it affect the original behavior
* [ ] Has unit tests been added
* [x] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1519107856
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1521442245
run p1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1518667945
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1515609899
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1521652048
run p1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #18834:
URL: https://github.com/apache/doris/pull/18834#discussion_r1173346789
##########
docs/en/docs/lakehouse/multi-catalog/hive.md:
##########
@@ -56,10 +56,13 @@ CREATE CATALOG hive PROPERTIES (
In addition to `type` and `hive.metastore.uris` , which are required, you can specify other parameters regarding the connection.
-> `specified_database_list`:
->
-> only synchronize the specified databases, split with ','. Default values is '' will synchronize all databases. db name is case sensitive.
->
+> `include_database_list`:
Review Comment:
This is a common property, you can put it in this doc: https://doris.apache.org/zh-CN/docs/dev/lakehouse/multi-catalog/
So that you don't need to write it for every catalog type
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java:
##########
@@ -130,10 +130,15 @@ protected void init() {
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.HMS);
List<String> allDatabases = client.getAllDatabases();
- Map<String, Boolean> specifiedDatabaseMap = getSpecifiedDatabaseMap();
+ Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
+ Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
// Update the db name to id map.
for (String dbName : allDatabases) {
- if (!specifiedDatabaseMap.isEmpty() && specifiedDatabaseMap.get(dbName) == null) {
+ // Exclude database map take effect with higher priority over include database map
+ if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.get(dbName) != null) {
Review Comment:
same for others
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java:
##########
@@ -130,10 +130,15 @@ protected void init() {
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.HMS);
List<String> allDatabases = client.getAllDatabases();
- Map<String, Boolean> specifiedDatabaseMap = getSpecifiedDatabaseMap();
+ Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
+ Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
// Update the db name to id map.
for (String dbName : allDatabases) {
- if (!specifiedDatabaseMap.isEmpty() && specifiedDatabaseMap.get(dbName) == null) {
+ // Exclude database map take effect with higher priority over include database map
+ if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.get(dbName) != null) {
Review Comment:
```suggestion
if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.containsKey(dbName)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on a diff in pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on code in PR #18834:
URL: https://github.com/apache/doris/pull/18834#discussion_r1173850716
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java:
##########
@@ -130,10 +130,15 @@ protected void init() {
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.HMS);
List<String> allDatabases = client.getAllDatabases();
- Map<String, Boolean> specifiedDatabaseMap = getSpecifiedDatabaseMap();
+ Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
+ Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
// Update the db name to id map.
for (String dbName : allDatabases) {
- if (!specifiedDatabaseMap.isEmpty() && specifiedDatabaseMap.get(dbName) == null) {
+ // Exclude database map take effect with higher priority over include database map
+ if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.get(dbName) != null) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1517937011
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1521048945
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1514900776
@morningman @nextdreamblue please take a review, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on a diff in pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on code in PR #18834:
URL: https://github.com/apache/doris/pull/18834#discussion_r1173807309
##########
docs/en/docs/lakehouse/multi-catalog/hive.md:
##########
@@ -56,10 +56,13 @@ CREATE CATALOG hive PROPERTIES (
In addition to `type` and `hive.metastore.uris` , which are required, you can specify other parameters regarding the connection.
-> `specified_database_list`:
->
-> only synchronize the specified databases, split with ','. Default values is '' will synchronize all databases. db name is case sensitive.
->
+> `include_database_list`:
Review Comment:
OK, I'll move docs in hive.md and iceberg.md to multi-catalog.md.
And I think we should keep jdbc.md since config like `include_database_list` should work with `only_specified_database`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1519796756
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #18834:
URL: https://github.com/apache/doris/pull/18834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1514898843
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1520378441
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1517937066
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1521099191
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1517928019
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1532794526
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] WinkerDu commented on pull request #18834: [feature](multi-catalog) Rename multi-catalog config 'specified_database_list' to 'include_database_list', and introduce new multi-catalog config 'exclude_database_list'
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #18834:
URL: https://github.com/apache/doris/pull/18834#issuecomment-1519054410
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org