You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2021/08/30 08:40:48 UTC
[GitHub] [submarine] KUAN-HSUN-LI opened a new pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
KUAN-HSUN-LI opened a new pull request #731:
URL: https://github.com/apache/submarine/pull/731
### What is this PR for?
1. Create submarine model management database including `registered_models`, `registered_model_tags`, `model_versions` and `model_version_tags` tables.
2. Move the `params` and `metrics` tables from `submarine.sql` to `submarine-model.sql`
### What type of PR is it?
[Feature]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-1002
### How should this be tested?
run the sdk unit tests `pytest --cov=submarine -vs -m "not e2e"`
TestModelVersion and TestRegisteredModel are two relevant unit tests
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r702241818
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
Review comment:
It sounds better than BIGINT
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
Review comment:
It sounds better than BIGINT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on pull request #731:
URL: https://github.com/apache/submarine/pull/731#issuecomment-908159214
@jeff-901 help me review this PR. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] asfgit closed pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #731:
URL: https://github.com/apache/submarine/pull/731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] lowc1012 commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r702240821
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
Review comment:
It is recommended to use the singular name of the entity uniformly.
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
Review comment:
Please use DATETIME type instead of BIGINT
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
Review comment:
Use the DATETIME type uniformly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on pull request #731:
URL: https://github.com/apache/submarine/pull/731#issuecomment-913139662
@lowc1012 @pingsutw I have applied the singular table name and DATETIME type. Could you help me review the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on pull request #731:
URL: https://github.com/apache/submarine/pull/731#issuecomment-910076797
@jeff-901 I have added the foreign key to both metrics and params. Help me review the following code, thanks.
By the way, database operation will be done in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r702241783
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,80 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
Review comment:
I will change the table name to a singular name. The tests in Java will still use the plural name since it is testing mlflow tables and mlflow tables are using the plural names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r698917853
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,78 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
+ `step` bigint(11) NOT NULL COMMENT 'Step recorded for this metric entry: `BigInteger`.',
+ `is_nan` BOOLEAN NOT NULL COMMENT 'True if the value is in fact NaN.',
+ PRIMARY KEY (`id`, `key`, `timestamp`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `params`;
+CREATE TABLE `params` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT '`String` (limit 190 characters). Part of *Primary Key* for ``params`` table.',
+ `value` varchar(32) NOT NULL COMMENT '`String` (limit 190 characters). Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT '`String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ PRIMARY KEY (`id`, `key`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
It sounds great to add the foreign key constraint. I will add it. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r698917853
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,78 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
+ `step` bigint(11) NOT NULL COMMENT 'Step recorded for this metric entry: `BigInteger`.',
+ `is_nan` BOOLEAN NOT NULL COMMENT 'True if the value is in fact NaN.',
+ PRIMARY KEY (`id`, `key`, `timestamp`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `params`;
+CREATE TABLE `params` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT '`String` (limit 190 characters). Part of *Primary Key* for ``params`` table.',
+ `value` varchar(32) NOT NULL COMMENT '`String` (limit 190 characters). Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT '`String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ PRIMARY KEY (`id`, `key`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
It sounds great to add the foreign key constraint. I will add it. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jeff-901 commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
jeff-901 commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r698509511
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,78 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
+ `step` bigint(11) NOT NULL COMMENT 'Step recorded for this metric entry: `BigInteger`.',
+ `is_nan` BOOLEAN NOT NULL COMMENT 'True if the value is in fact NaN.',
+ PRIMARY KEY (`id`, `key`, `timestamp`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `params`;
+CREATE TABLE `params` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT '`String` (limit 190 characters). Part of *Primary Key* for ``params`` table.',
+ `value` varchar(32) NOT NULL COMMENT '`String` (limit 190 characters). Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT '`String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ PRIMARY KEY (`id`, `key`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
Do metrics and params id need to be foreign key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jeff-901 commented on a change in pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
jeff-901 commented on a change in pull request #731:
URL: https://github.com/apache/submarine/pull/731#discussion_r698509511
##########
File path: dev-support/database/submarine-model.sql
##########
@@ -0,0 +1,78 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+-- http://www.apache.org/licenses/LICENSE-2.0
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+DROP TABLE IF EXISTS `registered_models`;
+CREATE TABLE `registered_models` (
+ `name` VARCHAR(256) NOT NULL,
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `description` VARCHAR(5000),
+ CONSTRAINT `registered_models_pk` PRIMARY KEY (`name`),
+ UNIQUE (`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `registered_model_tags`;
+CREATE TABLE `registered_model_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `registered_model_tag_pk` PRIMARY KEY (`name`, `tag`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_versions`;
+CREATE TABLE `model_versions` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `user_id` VARCHAR(64) NOT NULL COMMENT 'Id of the created user',
+ `experiment_id` VARCHAR(64) NOT NULL,
+ `current_stage` VARCHAR(20) COMMENT 'Model stage ex: None, production...',
+ `creation_time` BIGINT,
+ `last_updated_time` BIGINT,
+ `source` VARCHAR(512) COMMENT 'Model saved link',
+ `dataset` VARCHAR(256) COMMENT 'Which dataset is used',
+ `description` VARCHAR(5000),
+ CONSTRAINT `model_version_pk` PRIMARY KEY (`name`, `version`),
+ FOREIGN KEY(`name`) REFERENCES `registered_models` (`name`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `model_version_tags`;
+CREATE TABLE `model_version_tags` (
+ `name` VARCHAR(256) NOT NULL,
+ `version` INTEGER NOT NULL,
+ `tag` VARCHAR(256) NOT NULL,
+ CONSTRAINT `model_version_tag_pk` PRIMARY KEY (`name`, `version`, `tag`),
+ FOREIGN KEY(`name`, `version`) REFERENCES `model_versions` (`name`, `version`) ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `metrics`;
+CREATE TABLE `metrics` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT 'Metric key: `String` (limit 190 characters). Part of *Primary Key* for ``metrics`` table.',
+ `value` float NOT NULL COMMENT 'Metric value: `Float`. Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT 'Metric worker_index: `String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ `timestamp` bigint(20) NOT NULL COMMENT 'Timestamp recorded for this metric entry: `BigInteger`. Part of *Primary Key* for ``metrics`` table.',
+ `step` bigint(11) NOT NULL COMMENT 'Step recorded for this metric entry: `BigInteger`.',
+ `is_nan` BOOLEAN NOT NULL COMMENT 'True if the value is in fact NaN.',
+ PRIMARY KEY (`id`, `key`, `timestamp`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+DROP TABLE IF EXISTS `params`;
+CREATE TABLE `params` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Experiment',
+ `key` varchar(190) NOT NULL COMMENT '`String` (limit 190 characters). Part of *Primary Key* for ``params`` table.',
+ `value` varchar(32) NOT NULL COMMENT '`String` (limit 190 characters). Defined as *Non-null* in schema.',
+ `worker_index` varchar(32) NOT NULL COMMENT '`String` (limit 32 characters). Part of *Primary Key* for\r\n ``metrics`` table.',
+ PRIMARY KEY (`id`, `key`, `worker_index`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
Do metrics and params id need to be foreign key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] KUAN-HSUN-LI commented on pull request #731: SUBMARINE-1002. Submarine model management database and database sdk
Posted by GitBox <gi...@apache.org>.
KUAN-HSUN-LI commented on pull request #731:
URL: https://github.com/apache/submarine/pull/731#issuecomment-910076797
@jeff-901 I have added the foreign key to both metrics and params. Help me review the following code, thanks.
By the way, database operation will be done in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org