[jira] [Comment Edited] (SUBMARINE-507) Submarine Environment Management

["Manikandan R (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117911#comment-17117911 ] 

Manikandan R edited comment on SUBMARINE-507 at 5/27/20, 4:47 PM:
------------------------------------------------------------------

Writing down my thoughts on storing environments..Please share your views.
  
 1. Following tables can be created in Submarine Metastore:
  
 a) Table Name: environments
  
 Columns:
  
 environment_id int primary key
 name varchar(255) unique not null
 description string
 location string
 docker_id int references docker_images(docker_id)
 kernel_id int  references kernel(kernel_id)
 created_date timestamp
 last_updated_date timestamp
  
 "location" column captures hdfs path of the environment file.
  
 b) Table Name: docker_images
  
 docker_id int primary key
 name varchar(255) unique not null
 description string
 created_date timestamp
 last_updated_date timestamp
  
 c) Table Name: kernel
  
 kernel_id int primary key
 name varchar(255) unique not null
 description string
 repository string
 repository_type enum(''private', 'public')
 created_date timestamp
 last_updated_date timestamp
  
 Having separate tables for docker_images and kernel give us lot of flexibility while operating environments. a) docker and kernel images could be created only once and used for many environments. b) Avoid creating the same images and kernel/conda again and again in registry and repository respectively.
 (If required, we can clean up these 2 tables if it grows very very big and becomes a bottleneck, but very unlikely).
  
 2. How to store environment file?
  
 Create a directory hdfs://mycluster/submarine/environments/ if it doesn't exists and use environment name as file name. For example, 
  
 hdfs://mycluster/submarine/environments/my_env.txt

We could possibly provide more options like HDFS in future phases.
  
 3. How to store docker_images?
  
 We could set up our own registry as part of starting up the server. Please refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details. There are several options for this storage as documented in [https://docs.docker.com/registry/configuration/#storage]. For first cut, We can begin with file system and can be iterated over next releases based on the need.
  
 4. How to store kernel/conda?
  
 There are 2 types. 1. Private 2. Public.
  
 For private repo, we will need to set up local repo's and can be used.


was (Author: manirajv06@gmail.com):
Writing down my thoughts on storing environments..Please share your views.
 
1. Following tables can be created in Submarine Metastore:
 
a) Table Name: environments
 
Columns:
 
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int  references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
 
"location" column captures hdfs path of the environment file.
 
b) Table Name: docker_images
 
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
 
c) Table Name: kernel
 
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
 
Having separate tables for docker_images and kernel give us lot of flexibility while operating environments. 
- docker and kernel images could be created only once and used for many environments.
- Avoid creating the same images and kernel/conda again and again in registry and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and becomes a bottleneck, but very unlikely).
 
2. How to store environment file?
 
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't exists and use environment name as file name.  For example, 
 
hdfs://mycluster/submarine/environments/my_env.txt
 
3. How to store docker_images?
 
We could set up our own registry as part of starting up the server. Please refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details. There are several options for this storage as documented in [https://docs.docker.com/registry/configuration/#storage]. For first cut, We can begin with file system and can be iterated over next releases based on the need.
 
4. How to store kernel/conda?
 
There are 2 types. 1. Private 2. Public.
 
For private repo, we will need to set up local repo's and can be used.

> Submarine Environment Management
> --------------------------------
>
>                 Key: SUBMARINE-507
>                 URL: https://issues.apache.org/jira/browse/SUBMARINE-507
>             Project: Apache Submarine
>          Issue Type: New Feature
>            Reporter: Manikandan R
>            Assignee: Manikandan R
>            Priority: Major
>              Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been persisted like experiments so that it can used for later use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org