You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2023/01/03 09:29:53 UTC
[GitHub] [doris] BePPPower opened a new pull request, #15570: [feature](multi-catalog) support postgresql jdbc catalog
BePPPower opened a new pull request, #15570:
URL: https://github.com/apache/doris/pull/15570
# Proposed changes
Issue Number: close #14527
support postgresql jdbc catalog
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [x] No Need
3. Has document been added or modified:
- [x] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [x] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15570:
URL: https://github.com/apache/doris/pull/15570#discussion_r1060545520
##########
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java:
##########
@@ -418,7 +441,72 @@ public Type mysqlTypeToDoris(JdbcFieldSchema fieldSchema) {
return ScalarType.createStringType();
default:
throw new JdbcClientException("Can not convert mysql data type to doris data type for type ["
- + mysqlType + "]");
+ + mysqlType + "]");
+ }
+ }
+
+ public Type postgresqlTypeToDoris(JdbcFieldSchema fieldSchema) {
+ String pgType = fieldSchema.getDataTypeName();
+ switch (pgType) {
+ case "int2":
+ case "smallserial":
+ return Type.SMALLINT;
+ case "int4":
+ case "serial":
+ return Type.INT;
+ case "int8":
+ case "bigserial":
+ return Type.BIGINT;
+ case "numeric": {
+ int precision = fieldSchema.getColumnSize() + 1;
Review Comment:
Oh, this is no longer need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15570:
URL: https://github.com/apache/doris/pull/15570#discussion_r1061556035
##########
docs/en/docs/ecosystem/external-table/multi-catalog.md:
##########
@@ -392,6 +394,28 @@ CREATE CATALOG jdbc PROPERTIES (
)
```
+**postgresql catalog example**
+
+```sql
+-- 1.2.0+ Version
+CREATE CATALOG pg_resource PROPERTIES (
+ "type"="jdbc",
+ "jdbc.user"="postgres",
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15570:
URL: https://github.com/apache/doris/pull/15570#issuecomment-1372145655
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #15570:
URL: https://github.com/apache/doris/pull/15570#discussion_r1061274596
##########
docs/en/docs/ecosystem/external-table/multi-catalog.md:
##########
@@ -392,6 +394,28 @@ CREATE CATALOG jdbc PROPERTIES (
)
```
+**postgresql catalog example**
+
+```sql
+-- 1.2.0+ Version
+CREATE CATALOG pg_resource PROPERTIES (
Review Comment:
```suggestion
CREATE RESOURCE pg_resource PROPERTIES (
```
##########
docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-CATALOG.md:
##########
@@ -142,6 +143,31 @@ CREATE CATALOG catalog_name PROPERTIES (
);
```
+ **postgresql**
+
+ ```sql
+ -- 1.2.0+ Version
+ CREATE CATALOG pg_resource PROPERTIES (
Review Comment:
```suggestion
CREATE RESOURCE pg_resource PROPERTIES (
```
##########
docs/en/docs/ecosystem/external-table/multi-catalog.md:
##########
@@ -392,6 +394,28 @@ CREATE CATALOG jdbc PROPERTIES (
)
```
+**postgresql catalog example**
+
+```sql
+-- 1.2.0+ Version
+CREATE CATALOG pg_resource PROPERTIES (
+ "type"="jdbc",
+ "jdbc.user"="postgres",
Review Comment:
the prefix `jdbc.` should be removed when creating resource.
It is different from `create catalog`
##########
docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md:
##########
@@ -464,6 +467,28 @@ CREATE CATALOG jdbc PROPERTIES (
)
```
+**POSTGRESQL catalog示例**
+
+```sql
+-- 1.2.0+ 版本
+CREATE CATALOG pg_resource PROPERTIES (
Review Comment:
```suggestion
CREATE RESOURCE pg_resource PROPERTIES (
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15570:
URL: https://github.com/apache/doris/pull/15570#issuecomment-1371017901
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15570:
URL: https://github.com/apache/doris/pull/15570#discussion_r1060546692
##########
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java:
##########
@@ -371,11 +395,10 @@ public Type mysqlTypeToDoris(JdbcFieldSchema fieldSchema) {
case "BIGINT":
return Type.BIGINT;
case "DATE":
- return ScalarType.getDefaultDateType(Type.DATE);
+ return Type.DATE;
Review Comment:
Because jdbc external table do not support convert other database type to DATEV2/TIMESTAMPV2 type of doris
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15570:
URL: https://github.com/apache/doris/pull/15570#issuecomment-1372145695
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15570:
URL: https://github.com/apache/doris/pull/15570#issuecomment-1372000551
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #15570:
URL: https://github.com/apache/doris/pull/15570#discussion_r1060541972
##########
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java:
##########
@@ -371,11 +395,10 @@ public Type mysqlTypeToDoris(JdbcFieldSchema fieldSchema) {
case "BIGINT":
return Type.BIGINT;
case "DATE":
- return ScalarType.getDefaultDateType(Type.DATE);
+ return Type.DATE;
Review Comment:
Why not still using `ScalarType.getDefaultDateType`, so that we can control the behavior by FE config?
##########
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java:
##########
@@ -418,7 +441,72 @@ public Type mysqlTypeToDoris(JdbcFieldSchema fieldSchema) {
return ScalarType.createStringType();
default:
throw new JdbcClientException("Can not convert mysql data type to doris data type for type ["
- + mysqlType + "]");
+ + mysqlType + "]");
+ }
+ }
+
+ public Type postgresqlTypeToDoris(JdbcFieldSchema fieldSchema) {
+ String pgType = fieldSchema.getDataTypeName();
+ switch (pgType) {
+ case "int2":
+ case "smallserial":
+ return Type.SMALLINT;
+ case "int4":
+ case "serial":
+ return Type.INT;
+ case "int8":
+ case "bigserial":
+ return Type.BIGINT;
+ case "numeric": {
+ int precision = fieldSchema.getColumnSize() + 1;
Review Comment:
Do we still need this `+1`? PG has no `unsigned` type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15570:
URL: https://github.com/apache/doris/pull/15570#issuecomment-1369694823
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.62 seconds
load time: 655 seconds
storage size: 17123668236 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230103120653_clickbench_pr_72938.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #15570: [feature](multi-catalog) support postgresql jdbc catalog
Posted by GitBox <gi...@apache.org>.
morningman merged PR #15570:
URL: https://github.com/apache/doris/pull/15570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org