You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2023/06/12 04:10:46 UTC

[GitHub] [kyuubi] pan3793 opened a new pull request, #4950: Migrate Kyuubi embedded database from Derby to SQLite

pan3793 opened a new pull request, #4950:
URL: https://github.com/apache/kyuubi/pull/4950

   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
     2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
   -->
   
   ### _Why are the changes needed?_
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about the use case of it.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
   
   SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
   
   SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file are allowed, which helps a lot to analyze the data when the Kyuubi server is running.
   
   ### _How was this patch tested?_
   - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226113011


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -37,7 +37,9 @@ object JDBCMetadataStoreConf {
   val METADATA_STORE_JDBC_DATABASE_TYPE: ConfigEntry[String] =
     buildConf("kyuubi.metadata.store.jdbc.database.type")
       .doc("The database type for server jdbc metadata store.<ul>" +
-        " <li>DERBY: Apache Derby, JDBC driver `org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>(Deprecated) DERBY: Apache Derby, JDBC driver " +
+        "`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +

Review Comment:
   Good to be left as it is. The SQLITE option is not shown on the docs of 1.7.x after all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] cfmcgrady commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226110834


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -37,7 +37,9 @@ object JDBCMetadataStoreConf {
   val METADATA_STORE_JDBC_DATABASE_TYPE: ConfigEntry[String] =
     buildConf("kyuubi.metadata.store.jdbc.database.type")
       .doc("The database type for server jdbc metadata store.<ul>" +
-        " <li>DERBY: Apache Derby, JDBC driver `org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>(Deprecated) DERBY: Apache Derby, JDBC driver " +
+        "`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +

Review Comment:
   shall we make a note in the documentation to indicate that the support for SQLite is available starting from version 1.8.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 commented on pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#issuecomment-1587299890

   Thanks, merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226064581


##########
kyuubi-server/src/main/resources/sql/sqlite/001-KYUUBI-3967.sqlite.sql:
##########
@@ -0,0 +1,47 @@
+SELECT '< KYUUBI-3967: Shorten column varchar length of metadata table >' AS ' ';
+
+BEGIN;

Review Comment:
   Unfortunately, SQLite does not support `ALTER TABLE MODIFY COLUMN`, it was explained at 
   https://www3.sqlite.org/matrix/lang_altertable.html
   
   > "Why ALTER TABLE is such a problem for SQLite"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 closed pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite
URL: https://github.com/apache/kyuubi/pull/4950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226113301


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala:
##########
@@ -37,7 +37,9 @@ object JDBCMetadataStoreConf {
   val METADATA_STORE_JDBC_DATABASE_TYPE: ConfigEntry[String] =
     buildConf("kyuubi.metadata.store.jdbc.database.type")
       .doc("The database type for server jdbc metadata store.<ul>" +
-        " <li>DERBY: Apache Derby, JDBC driver `org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>(Deprecated) DERBY: Apache Derby, JDBC driver " +
+        "`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
+        " <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +

Review Comment:
   We don't do that for other similar cases, and we provided versioned docs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226633437


##########
docs/deployment/migration-guide.md:
##########
@@ -17,6 +17,13 @@
 
 # Kyuubi Migration Guide
 
+## Upgrading from Kyuubi 1.7 to 1.8
+
+* Since Kyuubi 1.8, SQLite is added and becomes the default database type of Kyuubi metastore, as Derby has been deprecated.
+  Both of Derby and SQLite are mainly for test purpose, and they're not supposed to use in production

Review Comment:
   ```suggestion
     Both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] pan3793 commented on pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#issuecomment-1586547777

   cc @turboFei @bowenliang123 @yaooqinn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] codecov-commenter commented on pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#issuecomment-1586613039

   ## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#4950](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b988348) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/b10a8ef27d0d9256eddd17d1e8eba3464ea19f31?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b10a8ef) will **not change** coverage.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@          Coverage Diff           @@
   ##           master   #4950   +/-   ##
   ======================================
     Coverage    0.00%   0.00%           
   ======================================
     Files         563     563           
     Lines       30931   30935    +4     
     Branches     4032    4033    +1     
   ======================================
   - Misses      30931   30935    +4     
   ```
   
   
   | [Impacted Files](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [.../main/scala/org/apache/kyuubi/util/JdbcUtils.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS91dGlsL0pkYmNVdGlscy5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/kyuubi/server/metadata/jdbc/DatabaseType.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9EYXRhYmFzZVR5cGUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
   | [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
   | [...i/server/metadata/jdbc/JDBCMetadataStoreConf.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZUNvbmYuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
   | [...ubi/server/metadata/jdbc/JdbcDatabaseDialect.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4950?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KZGJjRGF0YWJhc2VEaWFsZWN0LnNjYWxh) | `0.00% <0.00%> (ø)` | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #4950: Migrate Kyuubi embedded database from Derby to SQLite

Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #4950:
URL: https://github.com/apache/kyuubi/pull/4950#discussion_r1226578502


##########
docs/deployment/migration-guide.md:
##########
@@ -17,6 +17,13 @@
 
 # Kyuubi Migration Guide
 
+## Upgrading from Kyuubi 1.7 to 1.8
+
+* Since Kyuubi 1.8, SQLite is added and becomes the default data type of Kyuubi metastore, Derby is deprecated.

Review Comment:
   ```suggestion
   * Since Kyuubi 1.8, SQLite is added and becomes the default database type of Kyuubi metastore, as Derby has been deprecated.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org