You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by li...@apache.org on 2020/07/06 01:46:47 UTC
[submarine] branch master updated: SUBMARINE-550. [doc] Upload
pysubmarine to pypi
This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 24d18c0 SUBMARINE-550. [doc] Upload pysubmarine to pypi
24d18c0 is described below
commit 24d18c0b7ad2a26c7b46c4d68196c92845f51077
Author: HUAN-PING SU <pi...@gmail.com>
AuthorDate: Sun Jul 5 23:00:43 2020 +0800
SUBMARINE-550. [doc] Upload pysubmarine to pypi
### What is this PR for?
The doc including how to upload python package to pypi
Note:
I've uploaded pysubmarine package to **testPypi** for testing
https://test.pypi.org/project/submarine-sdk/
`pip install -i https://test.pypi.org/simple/ submarine-sdk
`
After review, I will create an account and push the package to **Pypi**
After discuss offline, The final name will be `apache-submarine`
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-550
### How should this be tested?
https://travis-ci.org/github/pingsutw/hadoop-submarine/builds/703397771
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: HUAN-PING SU <pi...@gmail.com>
Author: pingsutw <pi...@gmail.com>
Closes #334 from pingsutw/pypi and squashes the following commits:
2b611ab [HUAN-PING SU] Fix typo
53370ac [pingsutw] Update
532c741 [HUAN-PING SU] Update submarine-sdk/pysubmarine/README.md
f5aed45 [HUAN-PING SU] Update submarine-sdk/pysubmarine/setup.py
fba1b40 [pingsutw] SUBMARINE-550. [doc] Upload pysubmarine to pypi
---
docs/submarine-sdk/README.md | 4 +-
docs/submarine-sdk/pysubmarine/development.md | 81 ++++++++++++++++++++++
docs/submarine-sdk/pysubmarine/tracking.md | 3 -
.../pysubmarine/README.md | 32 +++------
.../github-actions/pypi-requirements.txt | 19 +++++
submarine-sdk/pysubmarine/setup.py | 9 ++-
6 files changed, 118 insertions(+), 30 deletions(-)
diff --git a/docs/submarine-sdk/README.md b/docs/submarine-sdk/README.md
index dd4fc9f..207aadc 100644
--- a/docs/submarine-sdk/README.md
+++ b/docs/submarine-sdk/README.md
@@ -17,11 +17,9 @@
## Summary
- Support Python, Scala, R language for algorithm development
-- support tracking/metrics APIs which allows developers
+- Support tracking/metrics APIs which allows developers
add tracking/metrics and view tracking/metrics from Submarine Workbench UI.
-- [TODO] Support user building ML pipeline
-
### Python
- [pysubmarine](pysubmarine)
diff --git a/docs/submarine-sdk/pysubmarine/development.md b/docs/submarine-sdk/pysubmarine/development.md
new file mode 100644
index 0000000..29e5031
--- /dev/null
+++ b/docs/submarine-sdk/pysubmarine/development.md
@@ -0,0 +1,81 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Python Development
+This page provides general Python development guidelines and source build instructions
+### Prerequisites
+This is required for developing & testing changes, we recommend installing pysubmarine
+in its own conda environment by running the following
+```bash
+conda create --name submarine-dev python=3.6
+conda activate submarine-dev
+
+# lint-requirements.txt and test-requirements.txt are in ./submarine-sdk/pysubmarine/github-actions
+pip install -r lint-requirements.txt
+pip install -r test-requirements.txt
+
+# Installs pysubmarine from current checkout
+pip install ./submarine-sdk/pysubmarine
+```
+### Coding Style
+- Use [yapf](https://github.com/google/yapf) to format Python code
+- yapf style is configured in `.style.yapf` file
+- To autoformat code
+```bash
+./submarine-sdk/pysubmarine/github-actions/auto-format.sh
+```
+- Verify linter pass before submitting a pull request by running:
+```bash
+./submarine-sdk/pysubmarine/github-actions/lint.sh
+```
+### Unit Testing
+We are using [pytest](https://docs.pytest.org/en/latest/) to develop our unit test suite.
+After building the project (see below) you can run its unit tests like so:
+```bash
+cd submarine-sdk/pysubmarine
+pytest --cov=submarine -vs
+```
+### Generate python SDK from swagger
+We use [swagger-codegen](https://swagger.io/docs/open-source-tools/swagger-codegen/)
+to generate pysubmarine client API that used to communicate with submarine server.
+
+If change below files, please run `./dev-support/pysubmarine/gen-sdk.sh`
+to generate latest version of SDK.
+- [Bootstrap.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java)
+- [ExperimentRestApi.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/ExperimentRestApi.java)
+
+### Upload package to PyPi
+For Apache Submarine committer and PMCs to do a new release.
+1. Change the version from 0.x.x-SNAPSHOT to 0.x.x
+in [setup.py](https://github.com/apache/submarine/blob/master/submarine-sdk/pysubmarine/setup.py)
+2. Install Python packages
+```bash
+cd submarine-sdk/pysubmarine
+pip install -r github-actions/pypi-requirements.txt
+```
+3. Compiling Your Package
+
+It will create `build`, `dist`, and `project.egg.info`
+in your local directory
+```bash
+python setup.py bdist_wheel
+```
+4. Upload python package to TestPyPI for testing
+```bash
+python -m twine upload --repository testpypi dist/*
+```
+5. Upload python package to PyPi
+```bash
+python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
+```
diff --git a/docs/submarine-sdk/pysubmarine/tracking.md b/docs/submarine-sdk/pysubmarine/tracking.md
index 3697b93..7a2d8e7 100644
--- a/docs/submarine-sdk/pysubmarine/tracking.md
+++ b/docs/submarine-sdk/pysubmarine/tracking.md
@@ -18,10 +18,7 @@ data exchange, and task tracking capabilities to more efficiently improve the
development and execution of machine learning productivity
- Allow data scientist to track distributed ML job
- Support store ML parameters and metrics in Submarine-server
-- [TODO] Support store ML job output (e.g. csv,images)
- Support hdfs,S3 and mysql (Currently we only support mysql)
-- [TODO] (WEB) Metric tracking ui in workbench-web
-- [TODO] (WEB) Metric graphical display in workbench-web
## Quickstart
1. [Start mini-submarine](../../../dev-support/mini-submarine#run-mini-submarine-image)
diff --git a/docs/submarine-sdk/pysubmarine/README.md b/submarine-sdk/pysubmarine/README.md
similarity index 64%
rename from docs/submarine-sdk/pysubmarine/README.md
rename to submarine-sdk/pysubmarine/README.md
index bf9c72e..ba7e578 100644
--- a/docs/submarine-sdk/pysubmarine/README.md
+++ b/submarine-sdk/pysubmarine/README.md
@@ -21,39 +21,25 @@ tracking experiment metrics, parameters.
## Package setup
-- Clone repository
+- Install latest version of pysubmarine
```bash
git clone https://github.com/apache/submarine.git
cd submarine/submarine-sdk/pysubmarine
-```
-- Install pip package
-```bash
pip install .
```
-- Run tests
+- Install package from pypi
```bash
-pytest --cov=submarine -vs
-```
-- Auto format code
-```bash
-./submarine-sdk/pysubmarine/github-actions/auto-format.sh
-```
-- Run checkstyle
-```bash
-./submarine-sdk/pysubmarine/github-actions/lint.sh
-```
-## How to generate REST SDK from swagger
-```bash
-./dev-support/pysubmarine/gen-sdk.sh
+pip install apache-submarine
```
## Easy-to-use model trainers
-- [FM](../../../submarine-sdk/pysubmarine/example/tensorflow/fm)
-- [DeepFM](../../../submarine-sdk/pysubmarine/example/tensorflow/deepfm)
+- [FM](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/tensorflow/fm)
+- [DeepFM](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/tensorflow/deepfm)
## Submarine experiment management
Makes it easy to run distributed or non-distributed TensorFlow, PyTorch experiments on Kubernetes.
-- [mnist example](../../../submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
+- [mnist example](https://github.com/apache/submarine/tree/master/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
+
+## Development
+See [Python Development](https://github.com/apache/submarine/tree/master/docs/submarine-sdk/pysubmarine/development.md) in the documentation subproject.
-## PySubmarine API Reference
-- [Tracking](tracking.md)
diff --git a/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
new file mode 100644
index 0000000..6521365
--- /dev/null
+++ b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+pip==20.1.1
+setuptools==47.3.1
+wheel==0.34.2
+twine==3.1.1
diff --git a/submarine-sdk/pysubmarine/setup.py b/submarine-sdk/pysubmarine/setup.py
index e30fed8..667c65f 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -15,11 +15,15 @@
from setuptools import setup, find_packages
+with open('README.md') as f:
+ long_description = f.read()
setup(
- name='pysubmarine',
+ name='apache-submarine',
version='0.5.0-SNAPSHOT',
description="A python SDK for submarine",
+ long_description=long_description,
+ long_description_content_type="text/markdown",
url="https://github.com/apache/submarine",
packages=find_packages(exclude=['tests', 'tests.*']),
install_requires=[
@@ -43,4 +47,7 @@ setup(
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
],
+ license='Apache License, Version 2.0',
+ maintainer='Apache Submarine Community',
+ maintainer_email='dev@submarine.apache.org',
)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org