You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "tigrulya-exe (via GitHub)" <gi...@apache.org> on 2024/01/29 13:00:36 UTC
[PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
tigrulya-exe opened a new pull request, #6027:
URL: https://github.com/apache/kyuubi/pull/6027
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #5674
## Describe Your Solution ๐ง
Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
`org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite`
---
# Checklist ๐
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469567573
##########
kyuubi-server/pom.xml:
##########
@@ -272,6 +272,11 @@
<artifactId>sqlite-jdbc</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.postgresql</groupId>
+ <artifactId>postgresql</artifactId>
Review Comment:
the LICENSE-binary and NOTICE-binary need to be updated since this new jar will be shipped by the binary tarball
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#issuecomment-1914680166
The [dependency check](https://github.com/apache/kyuubi/actions/runs/7696465875/job/20971423901?pr=6027) failed, please follow the tips in the error message to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6027: [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
URL: https://github.com/apache/kyuubi/pull/6027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469596334
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -55,8 +55,8 @@ object JDBCMetadataStoreConf {
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
" <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
- " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver` " +
"(fallback `com.mysql.jdbc.Driver`).</li>" +
+ " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`." +
Review Comment:
```suggestion
" <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`.</li>" +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "tigrulya-exe (via GitHub)" <gi...@apache.org>.
tigrulya-exe commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469594370
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -55,6 +55,7 @@ object JDBCMetadataStoreConf {
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
" <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
+ " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver` " +
Review Comment:
Nice catch! Fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469562711
##########
kyuubi-server/src/main/resources/sql/postgresql/metadata-store-schema-1.9.0.postgresql.sql:
##########
@@ -0,0 +1,59 @@
+CREATE TABLE IF NOT EXISTS metadata(
+ key_id bigserial PRIMARY KEY,
+ identifier varchar(36) NOT NULL,
+ session_type varchar(32) NOT NULL,
+ real_user varchar(255) NOT NULL,
+ user_name varchar(255) NOT NULL,
+ ip_address varchar(128),
+ kyuubi_instance varchar(1024),
+ state varchar(128) NOT NULL,
+ resource varchar(1024),
+ class_name varchar(1024),
+ request_name varchar(1024),
+ request_conf text,
+ request_args text,
+ create_time bigint NOT NULL,
+ engine_type varchar(32) NOT NULL,
+ cluster_manager varchar(128),
+ engine_open_time bigint,
+ engine_id varchar(128),
+ engine_name text,
+ engine_url varchar(1024),
+ engine_state varchar(32),
+ engine_error text,
+ end_time bigint,
+ priority int NOT NULL DEFAULT 10,
+ peer_instance_closed boolean DEFAULT FALSE
+);
+
+COMMENT ON COLUMN metadata.key_id IS 'the auto increment key id';
Review Comment:
Emm... does PostgreSQL support inline COMMENT syntax?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "tigrulya-exe (via GitHub)" <gi...@apache.org>.
tigrulya-exe commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469572946
##########
kyuubi-server/src/main/resources/sql/postgresql/metadata-store-schema-1.9.0.postgresql.sql:
##########
@@ -0,0 +1,59 @@
+CREATE TABLE IF NOT EXISTS metadata(
+ key_id bigserial PRIMARY KEY,
+ identifier varchar(36) NOT NULL,
+ session_type varchar(32) NOT NULL,
+ real_user varchar(255) NOT NULL,
+ user_name varchar(255) NOT NULL,
+ ip_address varchar(128),
+ kyuubi_instance varchar(1024),
+ state varchar(128) NOT NULL,
+ resource varchar(1024),
+ class_name varchar(1024),
+ request_name varchar(1024),
+ request_conf text,
+ request_args text,
+ create_time bigint NOT NULL,
+ engine_type varchar(32) NOT NULL,
+ cluster_manager varchar(128),
+ engine_open_time bigint,
+ engine_id varchar(128),
+ engine_name text,
+ engine_url varchar(1024),
+ engine_state varchar(32),
+ engine_error text,
+ end_time bigint,
+ priority int NOT NULL DEFAULT 10,
+ peer_instance_closed boolean DEFAULT FALSE
+);
+
+COMMENT ON COLUMN metadata.key_id IS 'the auto increment key id';
Review Comment:
As far as I know, it [doesn't support](https://www.postgresql.org/docs/current/sql-createtable.html) inline column comments. I can turn these commands to regular `--SQL comments` like in SQLite script if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#issuecomment-1914680627
Overall LGTM, thanks for contributing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#issuecomment-1916105068
@tigrulya-exe thanks, merged to master, and I opened [a follow-up PR](https://github.com/apache/kyuubi/pull/6029) to address the license stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469578144
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -55,6 +55,7 @@ object JDBCMetadataStoreConf {
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
" <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
+ " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver` " +
Review Comment:
looks like it should be swapped with the next line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469576615
##########
kyuubi-server/src/main/resources/sql/postgresql/metadata-store-schema-1.9.0.postgresql.sql:
##########
@@ -0,0 +1,59 @@
+CREATE TABLE IF NOT EXISTS metadata(
+ key_id bigserial PRIMARY KEY,
+ identifier varchar(36) NOT NULL,
+ session_type varchar(32) NOT NULL,
+ real_user varchar(255) NOT NULL,
+ user_name varchar(255) NOT NULL,
+ ip_address varchar(128),
+ kyuubi_instance varchar(1024),
+ state varchar(128) NOT NULL,
+ resource varchar(1024),
+ class_name varchar(1024),
+ request_name varchar(1024),
+ request_conf text,
+ request_args text,
+ create_time bigint NOT NULL,
+ engine_type varchar(32) NOT NULL,
+ cluster_manager varchar(128),
+ engine_open_time bigint,
+ engine_id varchar(128),
+ engine_name text,
+ engine_url varchar(1024),
+ engine_state varchar(32),
+ engine_error text,
+ end_time bigint,
+ priority int NOT NULL DEFAULT 10,
+ peer_instance_closed boolean DEFAULT FALSE
+);
+
+COMMENT ON COLUMN metadata.key_id IS 'the auto increment key id';
Review Comment:
Okay, let's leave it, the current way is fine too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "tigrulya-exe (via GitHub)" <gi...@apache.org>.
tigrulya-exe commented on code in PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#discussion_r1469606151
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -55,8 +55,8 @@ object JDBCMetadataStoreConf {
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
" <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
- " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver` " +
"(fallback `com.mysql.jdbc.Driver`).</li>" +
+ " <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`." +
Review Comment:
My bad, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6027:
URL: https://github.com/apache/kyuubi/pull/6027#issuecomment-1914899096
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `3 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`d474768`)](https://app.codecov.io/gh/apache/kyuubi/commit/d474768d97e85218183c880528a992a2dc258229?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.10% compared to head [(`1116580`)](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.07%.
> Report is 1 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ubi/server/metadata/jdbc/JdbcDatabaseDialect.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KZGJjRGF0YWJhc2VEaWFsZWN0LnNjYWxh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6027 +/- ##
============================================
- Coverage 61.10% 61.07% -0.03%
Complexity 23 23
============================================
Files 623 623
Lines 37103 37140 +37
Branches 5029 5032 +3
============================================
+ Hits 22672 22685 +13
- Misses 11981 12010 +29
+ Partials 2450 2445 -5
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6027?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org