You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by li...@apache.org on 2020/07/06 01:46:47 UTC

[submarine] branch master updated: SUBMARINE-550. [doc] Upload pysubmarine to pypi

This is an automated email from the ASF dual-hosted git repository.

liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git


The following commit(s) were added to refs/heads/master by this push:
     new 24d18c0  SUBMARINE-550. [doc] Upload pysubmarine to pypi
24d18c0 is described below

commit 24d18c0b7ad2a26c7b46c4d68196c92845f51077
Author: HUAN-PING SU <pi...@gmail.com>
AuthorDate: Sun Jul 5 23:00:43 2020 +0800

    SUBMARINE-550. [doc] Upload pysubmarine to pypi
    
    ### What is this PR for?
    The doc including how to upload python package to pypi
    
    Note:
    I've uploaded pysubmarine package to **testPypi** for testing
    https://test.pypi.org/project/submarine-sdk/
    `pip install -i https://test.pypi.org/simple/ submarine-sdk
    `
    
    After review, I will create an account and push the package to **Pypi**
    
    After discuss offline, The final name will be `apache-submarine`
    
    ### What type of PR is it?
    [Documentation]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    https://issues.apache.org/jira/browse/SUBMARINE-550
    
    ### How should this be tested?
    https://travis-ci.org/github/pingsutw/hadoop-submarine/builds/703397771
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    Author: HUAN-PING SU <pi...@gmail.com>
    Author: pingsutw <pi...@gmail.com>
    
    Closes #334 from pingsutw/pypi and squashes the following commits:
    
    2b611ab [HUAN-PING SU] Fix typo
    53370ac [pingsutw] Update
    532c741 [HUAN-PING SU] Update submarine-sdk/pysubmarine/README.md
    f5aed45 [HUAN-PING SU] Update submarine-sdk/pysubmarine/setup.py
    fba1b40 [pingsutw] SUBMARINE-550. [doc] Upload pysubmarine to pypi
---
 docs/submarine-sdk/README.md                       |  4 +-
 docs/submarine-sdk/pysubmarine/development.md      | 81 ++++++++++++++++++++++
 docs/submarine-sdk/pysubmarine/tracking.md         |  3 -
 .../pysubmarine/README.md                          | 32 +++------
 .../github-actions/pypi-requirements.txt           | 19 +++++
 submarine-sdk/pysubmarine/setup.py                 |  9 ++-
 6 files changed, 118 insertions(+), 30 deletions(-)

diff --git a/docs/submarine-sdk/README.md b/docs/submarine-sdk/README.md
index dd4fc9f..207aadc 100644
--- a/docs/submarine-sdk/README.md
+++ b/docs/submarine-sdk/README.md
@@ -17,11 +17,9 @@
 ## Summary
 - Support Python, Scala, R language for algorithm development
 
-- support tracking/metrics APIs which allows developers 
+- Support tracking/metrics APIs which allows developers 
 add tracking/metrics and view tracking/metrics from Submarine Workbench UI.
 
-- [TODO] Support user building ML pipeline
-
 ### Python 
 - [pysubmarine](pysubmarine)
 
diff --git a/docs/submarine-sdk/pysubmarine/development.md b/docs/submarine-sdk/pysubmarine/development.md
new file mode 100644
index 0000000..29e5031
--- /dev/null
+++ b/docs/submarine-sdk/pysubmarine/development.md
@@ -0,0 +1,81 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+## Python Development
+This page provides general Python development guidelines and source build instructions
+### Prerequisites
+This is required for developing & testing changes, we recommend installing pysubmarine
+in its own conda environment by running the following
+```bash
+conda create --name submarine-dev python=3.6
+conda activate submarine-dev
+
+# lint-requirements.txt and test-requirements.txt are in ./submarine-sdk/pysubmarine/github-actions
+pip install -r lint-requirements.txt
+pip install -r test-requirements.txt
+
+# Installs pysubmarine from current checkout
+pip install ./submarine-sdk/pysubmarine
+```
+### Coding Style
+- Use [yapf](https://github.com/google/yapf) to format Python code
+- yapf style is configured in `.style.yapf` file
+- To autoformat code
+```bash
+./submarine-sdk/pysubmarine/github-actions/auto-format.sh
+```
+- Verify linter pass before submitting a pull request by running:
+```bash
+./submarine-sdk/pysubmarine/github-actions/lint.sh
+```
+### Unit Testing
+We are using [pytest](https://docs.pytest.org/en/latest/) to develop our unit test suite.
+After building the project (see below) you can run its unit tests like so:
+```bash
+cd submarine-sdk/pysubmarine
+pytest --cov=submarine -vs
+```
+### Generate python SDK from swagger
+We use [swagger-codegen](https://swagger.io/docs/open-source-tools/swagger-codegen/)
+to generate pysubmarine client API that used to communicate with submarine server.
+
+If change below files, please run `./dev-support/pysubmarine/gen-sdk.sh`
+to generate latest version of SDK.
+- [Bootstrap.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java)
+- [ExperimentRestApi.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/ExperimentRestApi.java)
+
+### Upload package to PyPi
+For Apache Submarine committer and PMCs to do a new release.
+1. Change the version from 0.x.x-SNAPSHOT to 0.x.x
+in [setup.py](https://github.com/apache/submarine/blob/master/submarine-sdk/pysubmarine/setup.py)
+2. Install Python packages
+```bash
+cd submarine-sdk/pysubmarine
+pip install -r github-actions/pypi-requirements.txt
+```
+3. Compiling Your Package
+
+It will create `build`, `dist`, and `project.egg.info`
+in your local directory
+```bash
+python setup.py bdist_wheel
+```
+4. Upload python package to TestPyPI for testing
+```bash
+python -m twine upload --repository testpypi dist/*
+```
+5. Upload python package to PyPi
+```bash
+python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
+```
diff --git a/docs/submarine-sdk/pysubmarine/tracking.md b/docs/submarine-sdk/pysubmarine/tracking.md
index 3697b93..7a2d8e7 100644
--- a/docs/submarine-sdk/pysubmarine/tracking.md
+++ b/docs/submarine-sdk/pysubmarine/tracking.md
@@ -18,10 +18,7 @@ data exchange, and task tracking capabilities to more efficiently improve the
 development and execution of machine learning productivity
 - Allow data scientist to track distributed ML job
 - Support store ML parameters and metrics in Submarine-server
-- [TODO] Support store ML job output (e.g. csv,images)
 - Support hdfs,S3 and mysql (Currently we only support mysql)
-- [TODO] (WEB) Metric tracking ui in workbench-web
-- [TODO] (WEB) Metric graphical display in workbench-web
 
 ## Quickstart
 1. [Start mini-submarine](../../../dev-support/mini-submarine#run-mini-submarine-image)
diff --git a/docs/submarine-sdk/pysubmarine/README.md b/submarine-sdk/pysubmarine/README.md
similarity index 64%
rename from docs/submarine-sdk/pysubmarine/README.md
rename to submarine-sdk/pysubmarine/README.md
index bf9c72e..ba7e578 100644
--- a/docs/submarine-sdk/pysubmarine/README.md
+++ b/submarine-sdk/pysubmarine/README.md
@@ -21,39 +21,25 @@ tracking experiment metrics, parameters.
 
 
 ## Package setup
-- Clone repository
+- Install latest version of pysubmarine
 ```bash
 git clone https://github.com/apache/submarine.git
 cd submarine/submarine-sdk/pysubmarine
-```
-- Install pip package
-```bash
 pip install .
 ```
-- Run tests
+- Install package from pypi
 ```bash
-pytest --cov=submarine -vs
-```
-- Auto format code
-```bash
-./submarine-sdk/pysubmarine/github-actions/auto-format.sh
-```
-- Run checkstyle
-```bash
-./submarine-sdk/pysubmarine/github-actions/lint.sh
-```
-## How to generate REST SDK from swagger
-```bash
-./dev-support/pysubmarine/gen-sdk.sh
+pip install apache-submarine
 ```
 
 ## Easy-to-use model trainers
-- [FM](../../../submarine-sdk/pysubmarine/example/tensorflow/fm)
-- [DeepFM](../../../submarine-sdk/pysubmarine/example/tensorflow/deepfm)
+- [FM](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/tensorflow/fm)
+- [DeepFM](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/tensorflow/deepfm)
 
 ## Submarine experiment management
 Makes it easy to run distributed or non-distributed TensorFlow, PyTorch experiments on Kubernetes.
-- [mnist example](../../../submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
+- [mnist example](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
+
+## Development
+See [Python Development](https://github.com/apache/submarine/tree/master/docs/submarine-sdk/pysubmarine/development.md) in the documentation subproject.
 
-## PySubmarine API Reference
-- [Tracking](tracking.md)
diff --git a/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
new file mode 100644
index 0000000..6521365
--- /dev/null
+++ b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+pip==20.1.1
+setuptools==47.3.1
+wheel==0.34.2
+twine==3.1.1
diff --git a/submarine-sdk/pysubmarine/setup.py b/submarine-sdk/pysubmarine/setup.py
index e30fed8..667c65f 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -15,11 +15,15 @@
 
 from setuptools import setup, find_packages
 
+with open('README.md') as f:
+    long_description = f.read()
 
 setup(
-    name='pysubmarine',
+    name='apache-submarine',
     version='0.5.0-SNAPSHOT',
     description="A python SDK for submarine",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
     url="https://github.com/apache/submarine",
     packages=find_packages(exclude=['tests', 'tests.*']),
     install_requires=[
@@ -43,4 +47,7 @@ setup(
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
     ],
+    license='Apache License, Version 2.0',
+    maintainer='Apache Submarine Community',
+    maintainer_email='dev@submarine.apache.org',
 )


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org