You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2020/06/22 01:29:14 UTC
[GitHub] [submarine] wangdatan commented on a change in pull request #316: SUBMARINE-528. Polish documents for 0.4.0 release
wangdatan commented on a change in pull request #316:
URL: https://github.com/apache/submarine/pull/316#discussion_r443274533
##########
File path: docs/userdocs/k8s/helm.md
##########
@@ -20,31 +20,69 @@ under the License.
# Deploy Submarine On K8s
-## Deploy Submarine using Helm Chart (Recommended)
+## Deploy Submarine Using Helm Chart (Recommended)
Submarine's Helm Chart will not only deploy Submarine Server, but also deploys TF Operator / PyTorch Operator (which will be used by Submarine Server to run TF/PyTorch jobs on K8s).
-### Create images
-submarine server
+
+### Install Helm
+See https://helm.sh/docs/intro/install/
+
+### Install Submarine
```bash
-./dev-support/docker-images/submarine/build.sh
+helm install submarine ./helm-charts/submarine
```
+This will install submarine in the "default" namespace.
+The images are from Docker hub `apache/submarine`. See `./helm-charts/submarine/values.yaml` for more details
-submarine database
+If we'd like use a different namespace like "submarine"
```bash
-./dev-support/docker-images/database/build.sh
+kubectl create namespace submarine
+helm install submarine ./helm-charts/submarine -n submarine
Review comment:
We need to add a validation step after this, ask user to run `kubectl get pods`
And make sure following PODs are all running before proceeding next step.
```
NAME READY STATUS RESTARTS AGE
pytorch-operator-54854bf847-x65nk 1/1 Running 0 5m
submarine-database-5f74f747d-dzmf6 1/1 Running 0 5m
submarine-server-6f449bc967-cqkkv 1/1 Running 0 5m
tf-job-operator-c9cd7ccbd-4dzcs 1/1 Running 0 5m
```
##########
File path: README.md
##########
@@ -64,11 +64,56 @@ _Theodore Levitt_ once said:
Like mentioned above, Submarine is targeted to bring Data-Scientist-friendly user-interfaces to make their life easier. Here're some examples of Submarine user-interfaces.
-<FIXME: Add/FIX more contents below>
+### Submit a distributed Tensorflow experiment via Submarine Python SDK
-<WIP>
+#### Run a Tensorflow Mnist experiment
+```python
+submarine_client = submarine.ExperimentClient(host='http://submarine:8080')
+
+environment = Environment(image='gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0')
+
+experiment_meta = ExperimentMeta(name='mnist',
+ namespace='default',
+ framework='Tensorflow',
+ cmd='python /var/tf_mnist/mnist_with_summaries.py'
+ ' --log_dir=/train/log --learning_rate=0.01'
+ ' --batch_size=150')
+ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
+ replicas=1)
+worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
+ replicas=1)
+
+experiment_spec = ExperimentSpec(meta=experiment_meta,
+ environment=environment,
+ spec={'Ps':ps_spec, 'Worker': worker_spec})
+experiment = submarine_client.create_experiment(experiment_spec=experiment_spec)
+```
+
+#### Query a specific experiment
+```python
+submarine_client.get_experiment(experiment['experimentId'])
+```
+
+#### Wait for finish
+
+```python
+submarine_client.wait_for_finish(experiment['experimentId'])
+```
+
+#### Get the experiment's log
+```python
+submarine_client.get_log(experiment['experimentId'])
+```
+
+#### Get all running experiment
+```python
+submarine_client.list_experiments(status='running')
+```
+
+For a quick-start, see [Submarine On K8s](docs/userdocs/k8s/README.md)
+
+For more API details, see [SDK experiment example](submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
Review comment:
I suggest to NOT point .ipynb file here, instead, just point to k8s/README.md to ask user to follow the step and try it.
##########
File path: docs/userdocs/k8s/README.md
##########
@@ -31,6 +31,26 @@ After you have an up-and-running K8s, you can follow [Submarine Helm Charts Guid
## Use Submarine
### Model training (experiment) on K8s
+
+#### With Submarine SDK (Recommended)
+
+- Install SDK
+```bash
+git clone https://github.com/apache/submarine.git
+cd submarine/submarine-sdk/pysubmarine
+pip install .
+```
+> Note that this assumes Python3.7+.
+> If it fails to install, you should use a new Python environment created by `Anoconda` or Python `virtualenv`
Review comment:
To make users can successfully try that, can you export your Anaconda environment, so user can import it?
And the above section (git clone .. pip install) should be based on the conda environment.
##########
File path: docs/userdocs/k8s/helm.md
##########
@@ -20,31 +20,69 @@ under the License.
# Deploy Submarine On K8s
-## Deploy Submarine using Helm Chart (Recommended)
+## Deploy Submarine Using Helm Chart (Recommended)
Submarine's Helm Chart will not only deploy Submarine Server, but also deploys TF Operator / PyTorch Operator (which will be used by Submarine Server to run TF/PyTorch jobs on K8s).
-### Create images
-submarine server
+
+### Install Helm
+See https://helm.sh/docs/intro/install/
+
+### Install Submarine
```bash
-./dev-support/docker-images/submarine/build.sh
+helm install submarine ./helm-charts/submarine
Review comment:
Can you mention we need to run this command from Submarine's source folder (assume user will download release tarball, we should link to download page).
And we can mention if user want to try from source code, checkout a correct release branch.
##########
File path: docs/userdocs/k8s/helm.md
##########
@@ -20,31 +20,69 @@ under the License.
# Deploy Submarine On K8s
-## Deploy Submarine using Helm Chart (Recommended)
+## Deploy Submarine Using Helm Chart (Recommended)
Submarine's Helm Chart will not only deploy Submarine Server, but also deploys TF Operator / PyTorch Operator (which will be used by Submarine Server to run TF/PyTorch jobs on K8s).
-### Create images
-submarine server
+
+### Install Helm
+See https://helm.sh/docs/intro/install/
+
+### Install Submarine
```bash
-./dev-support/docker-images/submarine/build.sh
+helm install submarine ./helm-charts/submarine
Review comment:
Also, it's better to list what's the version of TF/PyTorch installed. This is important and users will ask.
And how we can support different TF/Pytorch versions? It's fine we don't support for now, but we should at least file a ticket in the community to track it if we don't have it for now.
##########
File path: README.md
##########
@@ -64,11 +64,56 @@ _Theodore Levitt_ once said:
Like mentioned above, Submarine is targeted to bring Data-Scientist-friendly user-interfaces to make their life easier. Here're some examples of Submarine user-interfaces.
-<FIXME: Add/FIX more contents below>
+### Submit a distributed Tensorflow experiment via Submarine Python SDK
-<WIP>
+#### Run a Tensorflow Mnist experiment
+```python
+submarine_client = submarine.ExperimentClient(host='http://submarine:8080')
+
+environment = Environment(image='gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0')
+
+experiment_meta = ExperimentMeta(name='mnist',
+ namespace='default',
+ framework='Tensorflow',
+ cmd='python /var/tf_mnist/mnist_with_summaries.py'
+ ' --log_dir=/train/log --learning_rate=0.01'
+ ' --batch_size=150')
Review comment:
It's better to add comments to these steps so user can understand what does this mean.
##########
File path: docs/userdocs/k8s/README.md
##########
@@ -31,6 +31,26 @@ After you have an up-and-running K8s, you can follow [Submarine Helm Charts Guid
## Use Submarine
### Model training (experiment) on K8s
+
+#### With Submarine SDK (Recommended)
+
+- Install SDK
+```bash
+git clone https://github.com/apache/submarine.git
Review comment:
Assume user download release tarball, please point to both downloads page as well as Github (and ask user to checkout a correct branch instead of master).
##########
File path: docs/userdocs/k8s/README.md
##########
@@ -31,6 +31,26 @@ After you have an up-and-running K8s, you can follow [Submarine Helm Charts Guid
## Use Submarine
### Model training (experiment) on K8s
+
+#### With Submarine SDK (Recommended)
+
+- Install SDK
+```bash
+git clone https://github.com/apache/submarine.git
+cd submarine/submarine-sdk/pysubmarine
+pip install .
+```
+> Note that this assumes Python3.7+.
+> If it fails to install, you should use a new Python environment created by `Anoconda` or Python `virtualenv`
+
+- Play with SDK
+
+Assuming you've installed submarine on K8s and forward the service to localhost, now you can open a Python shell, Jupyter notebook or any tools using the same Python
Review comment:
"Using the same Python environment" is ambiguous, since it is already mentioned above, I think we don't have to mention it here.
##########
File path: docs/userdocs/k8s/helm.md
##########
@@ -20,31 +20,69 @@ under the License.
# Deploy Submarine On K8s
-## Deploy Submarine using Helm Chart (Recommended)
+## Deploy Submarine Using Helm Chart (Recommended)
Submarine's Helm Chart will not only deploy Submarine Server, but also deploys TF Operator / PyTorch Operator (which will be used by Submarine Server to run TF/PyTorch jobs on K8s).
-### Create images
-submarine server
+
+### Install Helm
+See https://helm.sh/docs/intro/install/
+
+### Install Submarine
```bash
-./dev-support/docker-images/submarine/build.sh
+helm install submarine ./helm-charts/submarine
```
+This will install submarine in the "default" namespace.
+The images are from Docker hub `apache/submarine`. See `./helm-charts/submarine/values.yaml` for more details
-submarine database
+If we'd like use a different namespace like "submarine"
```bash
-./dev-support/docker-images/database/build.sh
+kubectl create namespace submarine
+helm install submarine ./helm-charts/submarine -n submarine
```
-### install helm
-For more info see https://helm.sh/docs/intro/install/
-### Deploy Submarine Server, mysql
-You can modify some settings in ./helm-charts/submarine/values.yaml
+> Note that if you encounter below issue when installation:
```bash
-helm install submarine ./helm-charts/submarine
+Error: rendered manifests contain a resource that already exists.
+Unable to continue with install: existing resource conflict: namespace: , name: podgroups.scheduling.incubator.k8s.io, existing_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition, new_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition
+```
+It might be caused by the previous installed submarine charts. Fix it by running:
+```bash
+kubectl delete crd/tfjobs.kubeflow.org && kubectl delete crd/podgroups.scheduling.incubator.k8s.io && kubectl delete crd/pytorchjobs.kubeflow.org
+```
+
+### Access Submarine Server locally
+
+```bash
+kubectl port-forward svc/submarine-server 8080:8080 -n submarine
Review comment:
By default we use default namespace, so -n is only needed when namespace is chosen when submarine is installed to a diffferent namespace, it's better to add an explanation.
##########
File path: docs/userdocs/k8s/README.md
##########
@@ -31,6 +31,26 @@ After you have an up-and-running K8s, you can follow [Submarine Helm Charts Guid
## Use Submarine
### Model training (experiment) on K8s
+
+#### With Submarine SDK (Recommended)
+
+- Install SDK
+```bash
+git clone https://github.com/apache/submarine.git
+cd submarine/submarine-sdk/pysubmarine
+pip install .
+```
+> Note that this assumes Python3.7+.
+> If it fails to install, you should use a new Python environment created by `Anoconda` or Python `virtualenv`
+
+- Play with SDK
Review comment:
"Play" is too causal. Suggest renaming it to "Run with Submarine Python SDK"
##########
File path: docs/userdocs/k8s/helm.md
##########
@@ -20,31 +20,69 @@ under the License.
# Deploy Submarine On K8s
-## Deploy Submarine using Helm Chart (Recommended)
+## Deploy Submarine Using Helm Chart (Recommended)
Submarine's Helm Chart will not only deploy Submarine Server, but also deploys TF Operator / PyTorch Operator (which will be used by Submarine Server to run TF/PyTorch jobs on K8s).
-### Create images
-submarine server
+
+### Install Helm
Review comment:
We should mention a minimum version of Helm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org