You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by pi...@apache.org on 2021/07/26 13:29:10 UTC
[submarine] branch master updated: SUBMARINE-938. [User] Modify
user documentation
This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 6eb2eb1 SUBMARINE-938. [User] Modify user documentation
6eb2eb1 is described below
commit 6eb2eb115fcdcf33b3d088ac767321acc7714c51
Author: ByronHsu <by...@gmail.com>
AuthorDate: Wed Jul 14 16:23:28 2021 +0800
SUBMARINE-938. [User] Modify user documentation
### What is this PR for?
1. Delete unimportant information and add critical ones.
2. Make the style consistent in the subpages.
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-938
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: ByronHsu <by...@gmail.com>
Signed-off-by: Kevin <pi...@apache.org>
Closes #678 from ByronHsu/modify-user-page and squashes the following commits:
dfe6f724 [ByronHsu] fix typo
c3ee1aa5 [ByronHsu] revert yarn
01c1269d [ByronHsu] revise user doc
---
dev-support/examples/quickstart/post.sh | 39 ++++++
dev-support/examples/tracking/tracking.py | 10 +-
website/docs/userDocs/api/environment.md | 6 +-
website/docs/userDocs/api/experiment-template.md | 10 +-
website/docs/userDocs/api/experiment.md | 22 +--
website/docs/userDocs/api/notebook.md | 2 +-
.../docs/userDocs/k8s/how-to-use-tensorboard.md | 64 ---------
.../userDocs/k8s/run-experiment-template-rest.md | 58 --------
.../userDocs/k8s/run-pytorch-experiment-rest.md | 148 ---------------------
.../userDocs/k8s/run-tensorflow-experiment-rest.md | 147 --------------------
website/docs/userDocs/others/mlflow.md | 24 +++-
.../userDocs/submarine-sdk/experiment-client.md | 81 ++---------
.../docs/userDocs/submarine-sdk/model-client.md | 139 +++++++++----------
.../userDocs/submarine-sdk/pysubmarine/tracking.md | 69 ----------
website/docs/userDocs/submarine-sdk/tracking.md | 4 -
15 files changed, 165 insertions(+), 658 deletions(-)
diff --git a/dev-support/examples/quickstart/post.sh b/dev-support/examples/quickstart/post.sh
new file mode 100644
index 0000000..39336bc
--- /dev/null
+++ b/dev-support/examples/quickstart/post.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+curl -X POST -H "Content-Type: application/json" -d '
+{
+ "meta": {
+ "name": "quickstart",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /opt/train.py",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "image": "quickstart:0.6.0-SNAPSHOT"
+ },
+ "spec": {
+ "Worker": {
+ "replicas": 3,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
+' http://127.0.0.1:32080/api/v1/experiment
\ No newline at end of file
diff --git a/dev-support/examples/tracking/tracking.py b/dev-support/examples/tracking/tracking.py
index 641c25e..61c9ad8 100644
--- a/dev-support/examples/tracking/tracking.py
+++ b/dev-support/examples/tracking/tracking.py
@@ -20,10 +20,10 @@ import random
import time
if __name__ == "__main__":
- periscope = ModelsClient()
- with periscope.start() as run:
- periscope.log_param("learning_rate", random.random())
+ modelClient = ModelsClient()
+ with modelClient.start() as run:
+ modelClient.log_param("learning_rate", random.random())
for i in range(100):
time.sleep(1)
- periscope.log_metric("mse", random.random() * 100, i)
- periscope.log_metric("acc", random.random(), i)
\ No newline at end of file
+ modelClient.log_metric("mse", random.random() * 100, i)
+ modelClient.log_metric("acc", random.random(), i)
\ No newline at end of file
diff --git a/website/docs/userDocs/api/environment.md b/website/docs/userDocs/api/environment.md
index 8559aed..851024b 100644
--- a/website/docs/userDocs/api/environment.md
+++ b/website/docs/userDocs/api/environment.md
@@ -45,9 +45,9 @@ Put EnvironmentSpec in request body.
| Field Name | Type | Description |
| ----------------- | -------------- | ---------------------------------- |
| name | String | Kernel name. |
-| channels | List\<String\> | Names of the channels. |
-| condaDependencies | List\<String\> | List of kernel conda dependencies. |
-| pipDependencies | List\<String\> | List of kernel pip dependencies. |
+| channels | List<String\> | Names of the channels. |
+| condaDependencies | List<String\> | List of kernel conda dependencies. |
+| pipDependencies | List<String\> | List of kernel pip dependencies. |
### Code Example
diff --git a/website/docs/userDocs/api/experiment-template.md b/website/docs/userDocs/api/experiment-template.md
index 274748d..76dbed5 100644
--- a/website/docs/userDocs/api/experiment-template.md
+++ b/website/docs/userDocs/api/experiment-template.md
@@ -34,7 +34,7 @@ POST /api/v1/template
| name | String | body | Experiment template name. This is required. |
| author | String | body | Author name. |
| description | String | body | Description of the experiment template. |
-| parameters | List\<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
+| parameters | List<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
| experimentSpec | ExperimentSpec | body | Spec of the experiment template. |
#### **ExperimentTemplateParamSpec**
@@ -52,7 +52,7 @@ POST /api/v1/template
| ----------- | --------------------------------- | --------------------------------------- |
| meta | ExperimentMeta | Meta data of the experiment template. |
| environment | EnvironmentSpec | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec\> | Spec of pods. |
| code | CodeSpec | Experiment codespec. |
#### **ExperimentMeta**
@@ -62,7 +62,7 @@ POST /api/v1/template
| namespace | String | Experiment namespace. |
| framework | String | Experiment framework. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
#### **EnvironmentSpec**
@@ -77,7 +77,7 @@ See more details in [environment api](https://submarine.apache.org/docs/userDocs
| name | String | Task name. |
| image | String | Image name. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
#### **CodeSpec**
@@ -413,7 +413,7 @@ PATCH /api/v1/template{name}
| name | String | path and body | Experiment template name. This is required. |
| author | String | body | Author name. |
| description | String | body | Description of the experiment template. |
-| parameters | List\<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
+| parameters | List<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
| experimentSpec | ExperimentSpec | body | Spec of the experiment template. |
### Code Example
diff --git a/website/docs/userDocs/api/experiment.md b/website/docs/userDocs/api/experiment.md
index e2e534a..87b8019 100644
--- a/website/docs/userDocs/api/experiment.md
+++ b/website/docs/userDocs/api/experiment.md
@@ -37,7 +37,7 @@ Put ExperimentSpec in request body.
| ----------- | --------------------------------- | --------------------------------------- |
| meta | ExperimentMeta | Meta data of the experiment template. |
| environment | EnvironmentSpec | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec> | Spec of pods. |
| code | CodeSpec | Experiment codespec. |
#### **ExperimentMeta**
@@ -48,10 +48,14 @@ Put ExperimentSpec in request body.
| namespace | String | Experiment namespace. |
| framework | String | Experiemnt framework. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String> | Environmental variables. |
#### **EnvironmentSpec**
+There are two types of environment: Anonymous and Predefined.
+- Anonymous environment: only specify `dockerImage` in environment spec. The container will be built on the docker image.
+- Embedded environment: specify `name` in environment spec. The container will be built on the existing environment (including dockerImage and kernalSpec).
+
See more details in [environment api](https://submarine.apache.org/docs/userDocs/api/environment).
#### **ExperimentTaskSpec**
@@ -63,14 +67,16 @@ See more details in [environment api](https://submarine.apache.org/docs/userDocs
| name | String | Task name. |
| image | String | Image name. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String> | Environmental variables. |
#### **CodeSpec**
-| Field Name | Type | Description |
-| ---------- | ------ | ----------------------- |
-| syncMode | String | sync mode of code spec. |
-| url | String | url of code spec. |
+Currently only support pulling from github. HDFS, NFS and s3 are in development
+
+| Field Name | Type | Description |
+| ---------- | ------------------------------| ----------------------- |
+| syncMode | String \(git\|hdfs\|nfs\|s3\) | sync mode of code spec. |
+| url | String | url of code spec. |
### Code Example
@@ -479,7 +485,7 @@ PATCH /api/v1/experiment/{id}
| id | String | path | Experiment id. |
| meta | ExperimentMeta | body | Meta data of the experiment template. |
| environment | EnvironmentSpec | body | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | body | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec> | body | Spec of pods. |
| code | CodeSpec | body | TODO |
### Code Example
diff --git a/website/docs/userDocs/api/notebook.md b/website/docs/userDocs/api/notebook.md
index f3e2ee6..1c9eb43 100644
--- a/website/docs/userDocs/api/notebook.md
+++ b/website/docs/userDocs/api/notebook.md
@@ -55,7 +55,7 @@ See more details in [environment api](https://submarine.apache.org/docs/userDocs
| Field Name | Type | Description |
| ---------- | --------------------- | ------------------------ |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
| resources | String | Resourecs of the pod. |
### Code Example
diff --git a/website/docs/userDocs/k8s/how-to-use-tensorboard.md b/website/docs/userDocs/k8s/how-to-use-tensorboard.md
deleted file mode 100644
index d5ef5cf..0000000
--- a/website/docs/userDocs/k8s/how-to-use-tensorboard.md
+++ /dev/null
@@ -1,64 +0,0 @@
----
-title: How to use tensorboard in Submarine
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Write to LogDirs by the environment variable
-
-### Environment variable
-
-- `SUBMARINE_TENSORBOARD_LOG_DIR`: Exist in every experiment container. You just need to direct your logs to `$(SUBMARINE_TENSORBOARD_LOG_DIR)` (**NOTICE: it is `()` not `{}`**), and you can inspect the process on the tensorboard webpage.
-
-### Example
-
-```
-{
- "meta": {
- "name": "tensorflow-tensorboard-dist-mnist",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=$(SUBMARINE_TENSORBOARD_LOG_DIR) --learning_rate=0.01 --batch_size=20",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=512M"
- }
- }
-}
-
-```
-
-## Connect to the tensorboard webpaage
-
-1. Open the experiment page in the workbench, and Click the `TensorBoard` button.
-
-
-
-2. Inspect the process on tensorboard page.
-
-
diff --git a/website/docs/userDocs/k8s/run-experiment-template-rest.md b/website/docs/userDocs/k8s/run-experiment-template-rest.md
deleted file mode 100644
index d379aaa..0000000
--- a/website/docs/userDocs/k8s/run-experiment-template-rest.md
+++ /dev/null
@@ -1,58 +0,0 @@
----
-title: Run Experiment Template Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Template Spec
-
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-### Use existing experiment template to create a experiment
-
-`POST /api/v1/experiment/{template-name}`
-
-**Example Request:**
-
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "params": {
- "learning_rate":"0.01",
- "batch_size":"150",
- "experiment_name":"newexperiment1"
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment/tf-mnist
-```
-
-**Example Request:**
-
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "params": {
- "experiment_name":"new-pytorch-mnist"
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment/pytorch-mnist
-```
-
-Register experiment template and more info see [Experiment Template API Reference](api/experiment-template.md).
diff --git a/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md b/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md
deleted file mode 100644
index f3c0053..0000000
--- a/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md
+++ /dev/null
@@ -1,148 +0,0 @@
----
-title: Run PyTorch Experiment Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Spec
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-**YAML Format:**
-```yaml
-meta:
- name: pytorch-mnist-yaml
- namespace: default
- framework: PyTorch
- cmd: python /var/mnist.py --backend gloo
- envVars:
- ENV_1: ENV1
-environment:
- image: apache/submarine:pytorch-dist-mnist-1.0
-spec:
- Master:
- replicas: 1
- resources: cpu=1,memory=1024M
- Worker:
- replicas: 1
- resources: cpu=1,memory=1024M
-```
-
-**JSON Format:**
-```json
-{
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-
-```
-
-## Create Experiment by REST API
-`POST /api/v1/experiment`
-
-**Example Request**
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment
-```
-
-**Example Response:**
-```json
-{
- "status": "OK",
- "code": 200,
- "result": {
- "experimentId": "experiment_1592057447228_0002",
- "name": "mnist",
- "uid": "38e39dcd-77d4-11ea-8dbb-0242ac110003",
- "status": "Accepted",
- "acceptedTime": "2020-06-13T22:19:29.000+08:00",
- "spec": {
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
- }
- }
-}
-```
-
-More info see [Experiment API Reference](api/experiment.md).
diff --git a/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md b/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md
deleted file mode 100644
index ae1c304..0000000
--- a/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md
+++ /dev/null
@@ -1,147 +0,0 @@
----
-title: Run TensorFlow Experiment Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Spec
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-**YAML Format:**
-```yaml
-meta:
- name: "tf-mnist-yaml"
- namespace: "default"
- framework: "TensorFlow"
- cmd: "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150"
- envVars:
- ENV_1: "ENV1"
-environment:
- image: "apache/submarine:tf-mnist-with-summaries-1.0"
-spec:
- Ps:
- replicas: 1
- resources: "cpu=1,memory=1024M"
- Worker:
- replicas: 1
- resources: "cpu=1,memory=1024M"
-```
-
-**JSON Format:**
-```json
-{
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-```
-
-## Create Experiment by REST API
-`POST /api/v1/experiment`
-
-**Example Request**
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment
-```
-
-**Example Response:**
-```json
-{
- "status": "OK",
- "code": 200,
- "result": {
- "experimentId": "experiment_1592057447228_0001",
- "name": "tf-mnist-json",
- "uid": "28e39dcd-77d4-11ea-8dbb-0242ac110003",
- "status": "Accepted",
- "acceptedTime": "2020-06-13T22:59:29.000+08:00",
- "spec": {
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
- }
- }
-}
-```
-
-More info see [Experiment API Reference](api/experiment.md).
diff --git a/website/docs/userDocs/others/mlflow.md b/website/docs/userDocs/others/mlflow.md
index 6d1043c..0341b4f 100644
--- a/website/docs/userDocs/others/mlflow.md
+++ b/website/docs/userDocs/others/mlflow.md
@@ -29,11 +29,25 @@ the result in MLflow UI. Below is the example of the usage of MLflow
UI.
### Example
-1. `cd ./submarine/dev-support/examples/tracking`
-2. `eval $(minikube -p minikube docker-env)`
-3. `sh build.sh`(it will take few minutes)
-4. `sh post.sh`
-5. In the MLflow UI page, you can see the log_param and the log_metric
+1. Run the following code in the cluster
+
+```python
+from submarine import ModelsClient
+import random
+import time
+
+if __name__ == "__main__":
+ modelClient = ModelsClient()
+ with modelClient.start() as run:
+ modelClient.log_param("learning_rate", random.random())
+ for i in range(100):
+ time.sleep(1)
+ modelClient.log_metric("mse", random.random() * 100, i)
+ modelClient.log_metric("acc", random.random(), i)
+```
+
+2. In the MLflow UI page, you can see the log_param and the log_metric
result. You can also compare the training between different workers.
+
diff --git a/website/docs/userDocs/submarine-sdk/experiment-client.md b/website/docs/userDocs/submarine-sdk/experiment-client.md
index e238595..096a0a9 100644
--- a/website/docs/userDocs/submarine-sdk/experiment-client.md
+++ b/website/docs/userDocs/submarine-sdk/experiment-client.md
@@ -30,8 +30,8 @@ Client of a submarine server that creates and manages experients and logs.
Create an experiment.
> **Parameters**
- **experiment_spec**: Submarine experiment spec. More detailed information can be found at [Experiment API](https://submarine.apache.org/docs/userDocs/api/experiment).
-> Returns
- - The detailed info about the submarine experiment.
+
+> **Returns**: The detailed info about the submarine experiment.
Example
@@ -64,63 +64,14 @@ client.create_experiment({
})
```
-Output
-
-```python
-{
- 'experimentId': 'experiment_1626160071451_0008',
- 'name': 'tf-mnist-json', 'uid': '3513233e-33f2-4399-8fba-2a44ca2af730',
- 'status': 'Accepted',
- 'acceptedTime': '2021-07-13T21:29:33.000+08:00',
- 'createdTime': None,
- 'runningTime': None,
- 'finishedTime': None,
- 'spec': {
- 'meta': {
- 'name': 'tf-mnist-json',
- 'namespace': 'default',
- 'framework': 'TensorFlow',
- 'cmd': 'python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150',
- 'envVars': {'ENV_1': 'ENV1'}
- },
- 'environment': {
- 'name': None,
- 'dockerImage': None,
- 'kernelSpec': None,
- 'description': None,
- 'image': 'apache/submarine:tf-mnist-with-summaries-1.0'
- },
- 'spec': {
- 'Ps': {
- 'replicas': 1,
- 'resources': 'cpu=1,memory=1024M',
- 'name': None,
- 'image': None,
- 'cmd': None,
- 'envVars': None,
- 'resourceMap': {'memory': '1024M', 'cpu': '1'}
- },
- 'Worker': {
- 'replicas': 1,
- 'resources': 'cpu=1,memory=1024M',
- 'name': None,
- 'image': None,
- 'cmd': None,
- 'envVars': None,
- 'resourceMap': {'memory': '1024M', 'cpu': '1'}
- }
- },
- 'code': None
- }
-}
-```
-
### `patch_experiment(id: str, experiment_spec: json) -> dict`
Patch an experiment.
> **Parameters**
- **id**: Submarine experiment id.
- **experiment_spec**: Submarine experiment spec. More detailed information can be found at [Experiment API](https://submarine.apache.org/docs/userDocs/api/experiment).
+
+
> **Returns**
- The detailed info about the submarine experiment.
@@ -203,26 +154,21 @@ By default print all the logs of Pod.
- **id**: Submarine experiment id.
- **onlyMaster**: By default include pod log of "master" which might be Tensorflow PS/Chief or PyTorch master.
+> **Return**
+ - The info of pod logs
+
Example
```python
client.get_log("experiment_1626160071451_0009")
```
-Output
-
-```
-The logs of Pod tf-mnist-json-2-ps-0:
-
-The logs of Pod tf-mnist-json-2-worker-0:
-
-```
-
### `list_log(status: str) -> list[dict]`
List experiment log.
> **Parameters**
- **status**: Accepted, Created, Running, Succeeded, Deleted.
+
> **Returns**
- List of submarine experiment logs.
@@ -232,22 +178,13 @@ Example
logs = client.list_log("Succeeded")
```
-Output
-
-```python
-[{'experimentId': 'experiment_1626160071451_0009',
- 'logContent':
- [{'podName': 'tf-mnist-json-2-ps-0', 'podLog': []},
- {'podName': 'tf-mnist-json-2-worker-0', 'podLog': []}]
-}]
-```
-
### `wait_for_finish(id: str, polling_interval: Optional[int]=10) -> dict`
Waits until the experiment is finished or failed.
> **Parameters**
- **id**: Submarine experiment id.
- **polling_interval**: How many seconds between two polls for the status of the experiment.
+
> **Returns**
- Submarine experiment logs.
diff --git a/website/docs/userDocs/submarine-sdk/model-client.md b/website/docs/userDocs/submarine-sdk/model-client.md
index 150d3a1..9e928d2 100644
--- a/website/docs/userDocs/submarine-sdk/model-client.md
+++ b/website/docs/userDocs/submarine-sdk/model-client.md
@@ -21,122 +21,123 @@ specific language governing permissions and limitations
under the License.
-->
-The submarine ModelsClient provides a high-level API for managing and serving your models.
+## class ModelClient()
-## ModelsClient.start()->[Active Run](https://mlflow.org/docs/latest/_modules/mlflow/tracking/fluent.html#ActiveRun)
+The submarine ModelsClient provides a high-level API for logging metrics / parameters and managing models.
-1. Start a new Mlflow run
-2. Direct the logging of the artifacts and metadata to the Run named "worker_i" under Experiment "job_id"
-3. If in distributed training, worker and job id would be parsed from environment variable
-4. If in local traning, worker and job id will be generated.
- :return: Active Run
+### `ModelsClient(tracking_uri=None, registry_uri=None)->ModelsClient`
-## ModelsClient.log_param(key, value)->None
+Initialize a `ModelsClient` instance.
-Log parameter under the current run.
+> **Parameters**
+ - **tracking_uri**: If run in Submarine, you do not need to specify it. Otherwise, specify the external tracking_uri.
+ - **registry_uri**: If run in Submarine, you do not need to specify it. Otherwise, specify the external registry_uri.
-- ### Parameters
- - **key** – Parameter name (string)
- - **value** – Parameter value (string, but will be string-ified if not)
-- ### example
+> **Returns**
+ - ModelsClient instance
-```
+Example
+
+```python
from submarine import ModelsClient
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_param("learning_rate", 0.01)
+modelClient = ModelsClient(tracking_uri="0.0.0.0:4000", tracking_uri="0.0.0.0:5000")
```
+### `ModelsClient.start()->[Active Run]`
-## ModelsClient.log_params(params)->None
+For details of [Active Run](https://mlflow.org/docs/latest/_modules/mlflow/tracking/fluent.html#ActiveRun)
-Log a batch of params for the current run.
+Start a new Mlflow run, and direct the logging of the artifacts and metadata to the Run named "worker_i" under Experiment "job_id". If in distributed training, worker and job id would be parsed from environment variable. If in local traning, worker and job id will be generated.
-- ### Parameters
+> **Returns**
+ - Active Run
+
+### `ModelsClient.log_param(key, value)->None`
+
+Log parameter under the current run.
- - **params** – Dictionary of param_name: String -> value: (String, but will be string-ified if not)
+> **Parameters**
+ - **key** – Parameter name
+ - **value** – Parameter value
-- ### example
+Example
+```python
+from submarine import ModelsClient
+
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_param("learning_rate", 0.01)
```
+
+### `ModelsClient.log_params(params)->None`
+
+Log a batch of params for the current run.
+
+> **Parameters**
+ - **params** – Dictionary of param_name: String -> value
+
+Example
+
+```python
from submarine import ModelsClient
params = {"learning_rate": 0.01, "n_estimators": 10}
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_params(params)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_params(params)
```
-## ModelsClient.log_metric(self, key, value, step=None)->None
+### `ModelsClient.log_metric(self, key, value, step=None)->None`
Log a metric under the current run.
-- ### Parameters
-
+> **Parameters**
- **key** – Metric name (string).
- - **value** – Metric value (float). Note that some special values such as +/- Infinity may be replaced by other values depending on the store. For example, the SQLAlchemy store replaces +/- Infinity with max / min float values.
+ - **value** – Metric value (float).
- **step** – Metric step (int). Defaults to zero if unspecified.
-- ### example
+Example
-```
+```python
from submarine import ModelsClient
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_metric("mse", 2500.00)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_metric("mse", 2500.00)
```
-## ModelsClient.log_metrics(self, metrics, step=None)->None
+### `ModelsClient.log_metrics(self, metrics, step=None)->None`
Log multiple metrics for the current run.
-- ### Parameters
-
- - **metrics** – Dictionary of metric_name: String -> value: Float. Note that some special values such as +/- Infinity may be replaced by other values depending on the store. For example, sql based store may replace +/- Infinity with max / min float values.
+> **Parameters**
+ - **metrics** – Dictionary of metric_name: String -> value: Float.
- **step** – A single integer step at which to log the specified Metrics. If unspecified, each metric is logged at step zero.
-- ### example
+Example
-```
+```python
from submarine import ModelsClient
metrics = {"mse": 2500.00, "rmse": 50.00}
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_metrics(metrics)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_metrics(metrics)
```
-## ModelsClient.load_model(self, name, version)->[ mlflow.pyfunc.PyFuncModel](https://mlflow.org/docs/latest/_modules/mlflow/pyfunc.html#PyFuncModel)
-
-Load a model stored in models Python function format with specific name and version.
-
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
-
-## ModelsClient.update_model(self, name, new_name)->None
-
-Update registered model name.
-
-- ### Parameters
- - **name** – Name of the registered model to update(string).
- - **new name** – New proposed name for the registered model(string).
-
-## ModelsClient.delete_model(self, name, version)->None
+### `(Beta) ModelsClient.save_model(self, model_type, model, artifact_path, registered_model_name=None)`
-Delete model version in backend.
+Save model to model registry.
+### `(Beta) ModelsClient.load_model(self, name, version)->mlflow.pyfunc.PyFuncModel`
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
+Load a model from model registry.
+### `(Beta) ModelsClient.update_model(self, name, new_name)->None`
-## ModelsClient.save_model(self, model_type, model, artifact_path, registered_model_name=None)
+Update a model by new name.
-Beta: Save model to server for managment and servering.
+### `(Beta) ModelsClient.delete_model(self, name, version)->None`
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
+Delete a model in model registry.
diff --git a/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md b/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md
deleted file mode 100644
index 7ab6e01..0000000
--- a/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md
+++ /dev/null
@@ -1,69 +0,0 @@
----
-title: PySubmarine Tracking
----
-<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-It helps developers use submarine's internal data caching,
-data exchange, and task tracking capabilities to more efficiently improve the
-development and execution of machine learning productivity
-- Allow data scientist to track distributed ML experiemnt
-- Support store ML parameters and metrics in Submarine-server
-- Support hdfs, S3 and mysql (Currently we only support mysql)
-
-## Quickstart
-1. [Start mini-submarine](https://github.com/apache/submarine/tree/master/dev-support/mini-submarine#run-mini-submarine-image)
-
-2. [Start Mysql server in mini-submarine](https://github.com/apache/submarine/tree/master/dev-support/mini-submarine#run-workbench-server)
-
-3. Uncomment the log_param and log_metric in
-[mnist_distributed.py](https://github.com/apache/submarine/blob/master/dev-support/mini-submarine/submarine/mnist_distributed.py)
-
-4. Start Submarine experiment (e.g., run_submarine_mnist_tony.sh)
-
-## Functions
-### `submarine.get_tracking_uri()`
-
-return the tracking URI.
-
-### `submarine.set_tracking_uri(URI)`
-
-set the tracking URI. You can also set the
-SUBMARINE_TRACKING_URI environment variable to have Submarine find a URI from
-there. The URI should be database connection string.
-
-**Parameters**
-
-- URI - Submarine record data to Mysql server. The database URL
-is expected in the format ``<dialect>+<driver>://<username>:<password>@<host>:<port>/<database>``.
-By default it's `mysql+pymysql://submarine:password@localhost:3306/submarine`.
-More detail : [SQLAlchemy docs](https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls)
-
-### `submarine.log_metric(key, value, step=0)`
-
-logs a single key-value metric. The value must always be a number.
-
-**Parameters**
-- key - Metric name (string).
-- value - Metric value (float).
-- step - A single integer step at which to log the specified Metrics,
-by default it's 0.
-
-### `submarine.log_param(key, value)`
-
-logs a single key-value parameter. The key and value are both strings.
-
-**Parameters**
-- key - Parameter name (string).
-- value - Parameter value (string).
diff --git a/website/docs/userDocs/submarine-sdk/tracking.md b/website/docs/userDocs/submarine-sdk/tracking.md
index 887683d..afacf39 100644
--- a/website/docs/userDocs/submarine-sdk/tracking.md
+++ b/website/docs/userDocs/submarine-sdk/tracking.md
@@ -36,7 +36,6 @@ development and execution of machine learning productivity
Get the tracking URI. If none has been specified, check the environmental variables. If uri is still none, return the default submarine jdbc url.
> **Returns**
-
- The tracking URI.
### `submarine.set_tracking_uri(uri: str) -> None`
@@ -44,7 +43,6 @@ Get the tracking URI. If none has been specified, check the environmental variab
set the tracking URI. You can also set the SUBMARINE_TRACKING_URI environment variable to have Submarine find a URI from there. The URI should be database connection string.
> **Parameters**
-
- **uri** \- Submarine record data to Mysql server. The database URL is expected in the format ``<dialect>+<driver>://<username>:<password>@<host>:<port>/<database>``.
By default it's `mysql+pymysql://submarine:password@localhost:3306/submarine`.
More detail : [SQLAlchemy docs](https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls)
@@ -54,7 +52,6 @@ set the tracking URI. You can also set the SUBMARINE_TRACKING_URI environment va
log a single key-value parameter. The key and value are both strings.
> **Parameters**
-
- **key** - Parameter name.
- **value** - Parameter value.
@@ -63,7 +60,6 @@ log a single key-value parameter. The key and value are both strings.
log a single key-value metric. The value must always be a number.
> **Parameters**
-
- **key** - Metric name.
- **value** - Metric value.
- **step** - A single integer step at which to log the specified Metrics, by default it's 0.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org