You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by pi...@apache.org on 2022/09/20 06:58:11 UTC
[submarine] branch master updated: SUBMARINE-1320. Drop supports for Python2.7 and Python3.6 (#994)
This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 7fa13a65 SUBMARINE-1320. Drop supports for Python2.7 and Python3.6 (#994)
7fa13a65 is described below
commit 7fa13a65530183c1ee424e14f7b92c3cbc1cd297
Author: Yao-Ching Huang <to...@gmail.com>
AuthorDate: Tue Sep 20 14:58:04 2022 +0800
SUBMARINE-1320. Drop supports for Python2.7 and Python3.6 (#994)
* refactor: cleanup
* feat: update linter configrations
* style: reformat by black
* docs: remove Python 3.6 support
* ci: remove Python3.6 from matrix
* feat: apply flake8 to dev-support/cicd
* fix: replace --experimental-string-processing with --preview
* style: fix python code style
* fix: ci requires types-certifi
* feat: update TF2 patch version & freeze keras to 2.6.0
* feat: update TF1 patch version & update matrix.tf-version
* feat: FsPreHandler is used by HDFSPreHandler
* ci: add unit-tf1 and unit-tf2
* ci: running `pip check` to verify compatible dependencies
* ci: fix package name
* feat: release numpy limitation
* test: add numpy
* test: update test requirements
* build: update dependency versions
* style: update
* feat: update pandas version
* feat: update pandas version
* ci: test tf1 with python3.7
* ci: add pip list
* ci: update
* feat: pin click
* fix: pin tensorflow-estimator
* fix: pin protobuf for tf1
* style: update
* ci: exclude partial combinations in matrix
* ci: update
* test: add e2e marker
* style: black
* feat: suppress `Cannot find implementation or library stub` error
* feat: pin importlib-metadata
* feat: suppress imported but unused error
* revert: aa21d5d4
* test: fix wrong params
* style: concate tuples
* feat: checkout rebased files
* ci: use compatible release operator to select the latest patch version
* feat: pin linter versions
* feat: update tf15 dependency version constraint
* docs: use Python3.6 for vsersion-0.0.7
* feat: update
* feat: prints out diffs
* ci: update
* ci: update
* ci: delete `pip check`
* feat: update flake8 config
* feat: update black arguments
* feat: update tf15 dependency constraint
* ci: simplify steps
* ci: update
* docs: update supported python versions
* style: format strings with f-string
* style: remove unnecessary f-string
* style: format strings with f-string
* feat: restore
---
.flake8 | 4 +-
.github/workflows/python.yml | 80 +++++++++++----
dev-support/cicd/merge_submarine_pr.py | 93 ++++++++---------
dev-support/database/init-database.py | 2 +-
.../mnist-pytorch/DDP/mnist_distributed.py | 43 +++-----
.../MirroredStrategy/mnist_keras_distributed.py | 10 +-
.../mnist_keras_distributed.py | 10 +-
.../mnist_keras_distributed.py | 7 +-
dev-support/examples/nn-pytorch/model.py | 2 +-
dev-support/examples/quickstart/train.py | 4 +-
.../submarine/image_classification.py | 64 ++++--------
.../mini-submarine/submarine/mnist_distributed.py | 13 +--
.../submarine/pytorch_mnist_distributed.py | 15 ++-
dev-support/style-check/python/auto-format.sh | 4 +-
.../style-check/python/lint-requirements.txt | 8 +-
dev-support/style-check/python/lint.sh | 4 +-
.../style-check/python/mypy-requirements.txt | 12 +--
.../scripts/combine-docker-daemons.py | 2 +-
pyproject.toml | 15 ++-
.../fs_prehandler/__init__.py | 2 +-
.../fs_prehandler/fs_prehandler.py | 1 +
.../fs_prehandler/hdfs_prehandler.py | 16 +--
submarine-experiment-prehandler/prehandler_main.py | 7 +-
.../pysubmarine/example/pytorch/afm/run_afm.py | 4 +-
.../example/pytorch/deepfm/run_deepfm.py | 4 +-
.../example/tensorflow/ccpm/run_ccpm.py | 4 +-
.../example/tensorflow/deepfm/run_deepfm.py | 4 +-
.../pysubmarine/example/tensorflow/fm/run_fm.py | 4 +-
submarine-sdk/pysubmarine/example/tracking.py | 2 +-
.../github-actions/pypi-requirements.txt | 2 +-
.../github-actions/test-requirements.txt | 30 +++---
submarine-sdk/pysubmarine/setup.py | 32 +++---
.../pysubmarine/submarine/artifacts/repository.py | 2 +-
.../pysubmarine/submarine/cli/__init__.py | 1 -
.../pysubmarine/submarine/cli/config/__init__.py | 7 +-
.../pysubmarine/submarine/cli/config/command.py | 10 +-
.../pysubmarine/submarine/cli/config/config.py | 10 +-
.../submarine/cli/environment/__init__.py | 6 +-
.../submarine/cli/environment/command.py | 24 +++--
.../submarine/cli/experiment/__init__.py | 6 +-
.../submarine/cli/experiment/command.py | 26 +++--
.../pysubmarine/submarine/cli/notebook/command.py | 24 +++--
.../pysubmarine/submarine/cli/sandbox/command.py | 2 +-
.../pysubmarine/submarine/cli/serve/__init__.py | 7 +-
.../pysubmarine/submarine/cli/serve/command.py | 15 +--
.../pysubmarine/submarine/client/__init__.py | 2 -
.../pysubmarine/submarine/client/api/__init__.py | 1 -
.../submarine/client/api/environment_api.py | 63 ++++--------
.../submarine/client/api/environment_client.py | 8 +-
.../submarine/client/api/experiment_api.py | 110 ++++++---------------
.../submarine/client/api/notebook_api.py | 61 ++++--------
.../pysubmarine/submarine/client/api/serve_api.py | 41 +++-----
.../pysubmarine/submarine/client/api_client.py | 40 +++-----
.../pysubmarine/submarine/client/configuration.py | 30 +++---
.../pysubmarine/submarine/client/exceptions.py | 27 +++--
.../submarine/client/models/__init__.py | 2 -
.../submarine/client/models/code_spec.py | 15 +--
.../submarine/client/models/environment_spec.py | 17 +---
.../submarine/client/models/experiment_meta.py | 17 +---
.../submarine/client/models/experiment_spec.py | 17 +---
.../client/models/experiment_task_spec.py | 15 +--
.../client/models/experiment_template_submit.py | 15 +--
.../submarine/client/models/json_response.py | 17 +---
.../submarine/client/models/kernel_spec.py | 15 +--
.../submarine/client/models/notebook_meta.py | 15 +--
.../submarine/client/models/notebook_pod_spec.py | 15 +--
.../submarine/client/models/notebook_spec.py | 19 +---
.../submarine/client/models/serve_request.py | 15 +--
.../submarine/client/models/serve_spec.py | 15 +--
submarine-sdk/pysubmarine/submarine/client/rest.py | 25 ++---
.../submarine/entities/_submarine_object.py | 8 +-
submarine-sdk/pysubmarine/submarine/exceptions.py | 10 +-
.../submarine/ml/pytorch/input/libsvm_dataset.py | 4 +-
.../submarine/ml/pytorch/layers/core.py | 10 +-
.../ml/pytorch/model/base_pytorch_model.py | 8 +-
.../submarine/ml/pytorch/model/ctr/afm.py | 10 +-
.../submarine/ml/pytorch/model/ctr/deepfm.py | 10 +-
.../submarine/ml/tensorflow/input/input.py | 2 +-
.../submarine/ml/tensorflow/layers/core.py | 19 ++--
.../submarine/ml/tensorflow/model/deepfm.py | 7 +-
.../submarine/ml/tensorflow/optimizer.py | 6 +-
.../submarine/ml/tensorflow_v2/input/input.py | 2 +-
.../submarine/ml/tensorflow_v2/layers/core.py | 20 ++--
.../submarine/ml/tensorflow_v2/model/fm.py | 6 +-
.../submarine/ml/tensorflow_v2/optimizer.py | 6 +-
.../pysubmarine/submarine/store/database/models.py | 28 ++----
.../store/model_registry/abstract_store.py | 4 +-
.../store/model_registry/sqlalchemy_store.py | 36 +++----
.../submarine/store/tracking/sqlalchemy_store.py | 2 +-
.../pysubmarine/submarine/tracking/client.py | 7 +-
.../pysubmarine/submarine/tracking/fluent.py | 1 -
.../pysubmarine/submarine/tracking/utils.py | 1 -
.../pysubmarine/submarine/utils/__init__.py | 4 +-
submarine-sdk/pysubmarine/submarine/utils/env.py | 2 +-
.../pysubmarine/submarine/utils/fileio.py | 8 +-
.../pysubmarine/submarine/utils/rest_utils.py | 7 +-
.../pysubmarine/submarine/utils/tf_utils.py | 15 +--
.../pysubmarine/submarine/utils/tf_utils_v2.py | 15 +--
.../pysubmarine/submarine/utils/validation.py | 28 +++---
submarine-sdk/pysubmarine/tests/cli/test_config.py | 16 ++-
.../pysubmarine/tests/cli/test_environment.py | 6 +-
.../pysubmarine/tests/cli/test_experiment.py | 10 +-
.../pysubmarine/tests/cli/test_notebook.py | 10 +-
.../pysubmarine/tests/cli/test_sandbox.py | 4 +-
.../tests/ml/tensorflow/model/test_deepfm.py | 1 -
.../tests/ml/tensorflow_v2/model/test_fm.py | 1 -
.../tests/ml/tensorflow_v2/test_optimizer.py | 1 -
.../tests/notebook/test_notebook_client.py | 4 +-
.../store/model_registry/test_sqlalchemy_store.py | 44 +++------
.../tests/store/tracking/test_sqlalchemy_store.py | 4 +-
.../pysubmarine/tests/tracking/test_tracking.py | 8 +-
.../pysubmarine/tests/tracking/test_utils.py | 3 +-
.../pysubmarine/tests/tracking/tf_model.py | 2 +-
.../pysubmarine/tests/utils/test_db_utils.py | 3 +-
.../pysubmarine/tests/utils/test_rest_utils.py | 6 +-
website/docs/devDocs/Dependencies.md | 9 +-
.../submarine-sdk/pysubmarine/development.md | 2 +-
117 files changed, 633 insertions(+), 1048 deletions(-)
diff --git a/.flake8 b/.flake8
index 0459d369..8f374063 100644
--- a/.flake8
+++ b/.flake8
@@ -16,5 +16,5 @@
#
[flake8]
-max-line-length = 100
-exclude = dev-support/cicd
\ No newline at end of file
+max-line-length = 110
+extend-ignore = E203 # https://black.readthedocs.io/en/stable/faq.html#why-are-flake8-s-e203-and-w503-violated
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index ac263a6d..a159f59c 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -26,12 +26,14 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
- name: Install dependencies
run: |
- pip install --upgrade pip
+ pip install --upgrade pip setuptools wheel
pip install -r ./dev-support/style-check/python/lint-requirements.txt
pip install -r ./dev-support/style-check/python/mypy-requirements.txt
+ - name: List installed packages
+ run: pip list
- name: Check python sdk code style
run: ./dev-support/style-check/python/lint.sh
@@ -40,33 +42,65 @@ jobs:
timeout-minutes: 30
strategy:
matrix:
- python-version: [3.6, 3.7]
- tf-version: [1.15.0, 2.6.0]
+ python-version: ['3.7', '3.8', '3.9', '3.10']
+ tf-version: ['1.15.0', '2.6.0', '2.7.0', '2.8.0', '2.9.0']
+ exclude:
+ - python-version: '3.8'
+ tf-version: '1.15.0'
+ - python-version: '3.9'
+ tf-version: '1.15.0'
+ - python-version: '3.10'
+ tf-version: '1.15.0'
+ - python-version: '3.10'
+ tf-version: '2.6.0'
+ - python-version: '3.10'
+ tf-version: '2.7.0'
fail-fast: false
steps:
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
- name: Setup python environment
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install requirements
run: |
- pip install --upgrade pip
+ pip install --upgrade pip setuptools wheel
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- name: Install pysubmarine with tf1 and pytorch
- if: matrix.tf-version == '1.15.0'
- run: pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf,pytorch]
+ if: ${{ matrix.tf-version == '1.15.0' }}
+ run: |
+ pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf,pytorch]
- name: Install pysubmarine with tf2 and pytorch
- if: matrix.tf-version == '2.6.0'
- run: pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf2,pytorch]
+ if: startsWith(matrix.tf-version, '2.')
+ run: |
+ pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf2,pytorch]
+ pip install --no-cache-dir 'tensorflow~=${{ matrix.tf-version }}'
+ - name: List installed packages
+ run: pip list
- name: Run unit test
run: pytest --cov=submarine -vs -m "not e2e"
integration:
runs-on: ubuntu-latest
timeout-minutes: 60
+ strategy:
+ matrix:
+ python-version: ['3.7', '3.8', '3.9', '3.10']
+ tf-version: ['1.15.0', '2.6.0', '2.7.0', '2.8.0', '2.9.0']
+ exclude:
+ - python-version: '3.8'
+ tf-version: '1.15.0'
+ - python-version: '3.9'
+ tf-version: '1.15.0'
+ - python-version: '3.10'
+ tf-version: '1.15.0'
+ - python-version: '3.10'
+ tf-version: '2.6.0'
+ - python-version: '3.10'
+ tf-version: '2.7.0'
+ fail-fast: false
steps:
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
with:
fetch-depth: 50
- name: Set up JDK 1.8
@@ -98,7 +132,7 @@ jobs:
echo "current-context:" $(kubectl config current-context)
echo "environment-kubeconfig:" ${KUBECONFIG}
# Cache maven (for submarine server)
- - uses: actions/cache@v2
+ - uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
@@ -109,16 +143,24 @@ jobs:
- name: Start submarine
run: bash ./.github/scripts/start-submarine.sh
- name: Setup python environment
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
- python-version: 3.6
- - name: Install dependencies
+ python-version: ${{ matrix.python-version }}
+ - name: Install tf1 dependencies
+ if: ${{ matrix.tf-version == '1.15.0' }}
run: |
- pip install --upgrade pip
+ pip install --upgrade pip setuptools wheel
pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf,pytorch]
- pip install --no-cache-dir tensorflow-addons
- pip install --no-cache-dir tf_slim
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+ - name: Install tf2 dependencies
+ if: startsWith(matrix.tf-version, '2.')
+ run: |
+ pip install --upgrade pip setuptools wheel
+ pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf2,pytorch]
+ pip install --no-cache-dir 'tensorflow~=${{ matrix.tf-version }}'
+ pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+ - name: List installed packages
+ run: pip list
- name: Set CLI config for e2e pytest
run: |
submarine config set connection.hostname localhost
diff --git a/dev-support/cicd/merge_submarine_pr.py b/dev-support/cicd/merge_submarine_pr.py
index c670f3fb..03979e3c 100755
--- a/dev-support/cicd/merge_submarine_pr.py
+++ b/dev-support/cicd/merge_submarine_pr.py
@@ -63,7 +63,7 @@ os.chdir(SUBMARINE_HOME)
def get_json(url):
try:
return json.load(urllib.request.urlopen(url))
- except urllib.error.HTTPError as e:
+ except urllib.error.HTTPError:
print("Unable to fetch URL, exiting: %s" % url)
sys.exit(-1)
@@ -104,10 +104,10 @@ def clean_up():
# merge the requested PR and return the merge hash
def merge_pr(pr_num, target_ref):
- pr_branch_name = "%s_MERGE_PR_%s" % (BRANCH_PREFIX, pr_num)
- target_branch_name = "%s_MERGE_PR_%s_%s" % (BRANCH_PREFIX, pr_num, target_ref.upper())
- run_cmd("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num, pr_branch_name))
- run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, target_ref, target_branch_name))
+ pr_branch_name = "{}_MERGE_PR_{}".format(BRANCH_PREFIX, pr_num)
+ target_branch_name = "{}_MERGE_PR_{}_{}".format(BRANCH_PREFIX, pr_num, target_ref.upper())
+ run_cmd("git fetch {} pull/{}/head:{}".format(PR_REMOTE_NAME, pr_num, pr_branch_name))
+ run_cmd("git fetch {} {}:{}".format(PUSH_REMOTE_NAME, target_ref, target_branch_name))
run_cmd("git checkout %s" % target_branch_name)
had_conflicts = False
@@ -119,17 +119,15 @@ def merge_pr(pr_num, target_ref):
msg = "Okay, please fix any conflicts and 'git add' conflicting files... Finished?"
continue_maybe(msg)
had_conflicts = True
- commit_authors = run_cmd(
- ["git", "log", "HEAD..%s" % pr_branch_name, "--pretty=format:%an <%ae>"]
- ).split("\n")
- commit_date = run_cmd(["git", "log", "%s" % pr_branch_name, "-1", "--pretty=format:%ad"])
- distinct_authors = sorted(
- set(commit_authors), key=lambda x: commit_authors.count(x), reverse=True
+ commit_authors = run_cmd(["git", "log", "HEAD..%s" % pr_branch_name, "--pretty=format:%an <%ae>"]).split(
+ "\n"
)
+ commit_date = run_cmd(["git", "log", "%s" % pr_branch_name, "-1", "--pretty=format:%ad"])
+ distinct_authors = sorted(set(commit_authors), key=lambda x: commit_authors.count(x), reverse=True)
primary_author = distinct_authors[0]
- commits = run_cmd(
- ["git", "log", "HEAD..%s" % pr_branch_name, "--pretty=format:%h [%an] %s"]
- ).split("\n\n")
+ commits = run_cmd(["git", "log", "HEAD..%s" % pr_branch_name, "--pretty=format:%h [%an] %s"]).split(
+ "\n\n"
+ )
merge_message_flags = []
@@ -147,11 +145,11 @@ def merge_pr(pr_num, target_ref):
committer_email = run_cmd("git config --get user.email").strip()
merge_message_flags += [
"-m",
- "\n" + "Signed-off-by: %s <%s>" % (committer_name, committer_email),
+ "\n" + "Signed-off-by: {} <{}>".format(committer_name, committer_email),
]
if had_conflicts:
- message = "This patch had conflicts when merged, resolved by\nCommitter: %s <%s>" % (
+ message = "This patch had conflicts when merged, resolved by\nCommitter: {} <{}>".format(
committer_name,
committer_email,
)
@@ -160,22 +158,19 @@ def merge_pr(pr_num, target_ref):
# The string "Closes #%s" string is required for GitHub to correctly close the PR
merge_message_flags += [
"-m",
- "Closes #%s from %s and squashes the following commits:" % (pr_num, pr_repo_desc),
+ "Closes #{} from {} and squashes the following commits:".format(pr_num, pr_repo_desc),
]
for c in commits:
merge_message_flags += ["-m", c]
run_cmd(
- ["git", "commit", '--author="%s"' % primary_author, '--date="%s"' % commit_date]
- + merge_message_flags
+ ["git", "commit", '--author="%s"' % primary_author, '--date="%s"' % commit_date] + merge_message_flags
)
- continue_maybe(
- "Merge complete (local ref %s). Push to %s?" % (target_branch_name, PUSH_REMOTE_NAME)
- )
+ continue_maybe("Merge complete (local ref {}). Push to {}?".format(target_branch_name, PUSH_REMOTE_NAME))
try:
- run_cmd("git push %s %s:%s" % (PUSH_REMOTE_NAME, target_branch_name, target_ref))
+ run_cmd("git push {} {}:{}".format(PUSH_REMOTE_NAME, target_branch_name, target_ref))
except Exception as e:
clean_up()
fail("Exception while pushing: %s" % e)
@@ -192,9 +187,9 @@ def cherry_pick(pr_num, merge_hash, default_branch):
if pick_ref == "":
pick_ref = default_branch
- pick_branch_name = "%s_PICK_PR_%s_%s" % (BRANCH_PREFIX, pr_num, pick_ref.upper())
+ pick_branch_name = "{}_PICK_PR_{}_{}".format(BRANCH_PREFIX, pr_num, pick_ref.upper())
- run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, pick_ref, pick_branch_name))
+ run_cmd("git fetch {} {}:{}".format(PUSH_REMOTE_NAME, pick_ref, pick_branch_name))
run_cmd("git checkout %s" % pick_branch_name)
try:
@@ -205,12 +200,10 @@ def cherry_pick(pr_num, merge_hash, default_branch):
msg = "Okay, please fix any conflicts and finish the cherry-pick. Finished?"
continue_maybe(msg)
- continue_maybe(
- "Pick complete (local ref %s). Push to %s?" % (pick_branch_name, PUSH_REMOTE_NAME)
- )
+ continue_maybe("Pick complete (local ref {}). Push to {}?".format(pick_branch_name, PUSH_REMOTE_NAME))
try:
- run_cmd("git push %s %s:%s" % (PUSH_REMOTE_NAME, pick_branch_name, pick_ref))
+ run_cmd("git push {} {}:{}".format(PUSH_REMOTE_NAME, pick_branch_name, pick_ref))
except Exception as e:
clean_up()
fail("Exception while pushing: %s" % e)
@@ -218,7 +211,7 @@ def cherry_pick(pr_num, merge_hash, default_branch):
pick_hash = run_cmd("git rev-parse %s" % pick_branch_name)[:8]
clean_up()
- print("Pull request #%s picked into %s!" % (pr_num, pick_ref))
+ print("Pull request #{} picked into {}!".format(pr_num, pick_ref))
print("Pick hash: %s" % pick_hash)
return pick_ref
@@ -233,9 +226,7 @@ def fix_version_from_branch(branch, versions) -> List:
def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
- asf_jira = jira.client.JIRA(
- {"server": JIRA_API_BASE}, basic_auth=(JIRA_USERNAME, JIRA_PASSWORD)
- )
+ asf_jira = jira.client.JIRA({"server": JIRA_API_BASE}, basic_auth=(JIRA_USERNAME, JIRA_PASSWORD))
jira_id = input("Enter a JIRA id [%s]: " % default_jira_id)
if jira_id == "":
@@ -244,7 +235,7 @@ def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
try:
issue = asf_jira.issue(jira_id)
except Exception as e:
- fail("ASF JIRA could not find %s\n%s" % (jira_id, e))
+ fail("ASF JIRA could not find {}\n{}".format(jira_id, e))
cur_status = issue.fields.status.name
cur_summary = issue.fields.summary
@@ -255,7 +246,7 @@ def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
cur_assignee = cur_assignee.displayName
if cur_status == "Resolved" or cur_status == "Closed":
- fail("JIRA issue %s already has status '%s'" % (jira_id, cur_status))
+ fail("JIRA issue {} already has status '{}'".format(jira_id, cur_status))
print("=== JIRA %s ===" % jira_id)
print(
"summary\t\t%s\nassignee\t%s\nstatus\t\t%s\nurl\t\t%s/%s\n"
@@ -266,7 +257,7 @@ def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
versions = sorted(versions, key=lambda x: x.name, reverse=True)
versions = filter(lambda x: x.raw["released"] is False, versions)
# Consider only x.y.z versions
- versions = filter(lambda x: re.match("\d+\.\d+\.\d+", x.name), versions)
+ versions = filter(lambda x: re.match(r"\d+\.\d+\.\d+", x.name), versions)
default_fix_versions: List = list(
map(lambda x: fix_version_from_branch(x, list(versions)).name, merge_branches)
@@ -278,7 +269,7 @@ def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
# both 1.1.0 and 1.0.0 as fix versions.
(major, minor, patch) = v.split(".")
if patch == "0":
- previous = "%s.%s.%s" % (major, int(minor) - 1, 0)
+ previous = "{}.{}.{}".format(major, int(minor) - 1, 0)
if previous in default_fix_versions:
default_fix_versions: List = list(filter(lambda x: x != v, default_fix_versions))
default_fix_versions = ",".join(default_fix_versions)
@@ -294,11 +285,9 @@ def resolve_jira_issue(merge_branches, comment, default_jira_id=""):
jira_fix_versions = list(map(lambda v: get_version_json(v), fix_versions))
resolve = list(filter(lambda a: a["name"] == "Resolve Issue", asf_jira.transitions(jira_id)))[0]
- asf_jira.transition_issue(
- jira_id, resolve["id"], fixVersions=jira_fix_versions, comment=comment
- )
+ asf_jira.transition_issue(jira_id, resolve["id"], fixVersions=jira_fix_versions, comment=comment)
- print("Successfully resolved %s with fixVersions=%s!" % (jira_id, fix_versions))
+ print("Successfully resolved {} with fixVersions={}!".format(jira_id, fix_versions))
def resolve_jira_issues(title, merge_branches, comment):
@@ -317,8 +306,8 @@ def resolve_jira_issues(title, merge_branches, comment):
latest_branch = "master"
pr_num = input("Which pull request would you like to merge? (e.g. 23): ")
-pr = get_json("%s/pulls/%s" % (GITHUB_API_BASE, pr_num))
-pr_events = get_json("%s/issues/%s/events" % (GITHUB_API_BASE, pr_num))
+pr = get_json("{}/pulls/{}".format(GITHUB_API_BASE, pr_num))
+pr_events = get_json("{}/issues/{}/events".format(GITHUB_API_BASE, pr_num))
url = pr["url"]
title = pr["title"]
@@ -326,7 +315,7 @@ body = pr["body"]
target_ref = pr["base"]["ref"]
user_login = pr["user"]["login"]
base_ref = pr["head"]["ref"]
-pr_repo_desc = "%s/%s" % (user_login, base_ref)
+pr_repo_desc = "{}/{}".format(user_login, base_ref)
# Merged pull requests don't appear as merged in the GitHub API;
# Instead, they're closed by asfgit.
@@ -334,29 +323,25 @@ merge_commits = [e for e in pr_events if e["actor"]["login"] == "asfgit" and e["
if merge_commits:
merge_hash = merge_commits[0]["commit_id"]
- message = get_json("%s/commits/%s" % (GITHUB_API_BASE, merge_hash))["commit"]["message"]
+ message = get_json("{}/commits/{}".format(GITHUB_API_BASE, merge_hash))["commit"]["message"]
print("Pull request %s has already been merged, assuming you want to backport" % pr_num)
commit_is_downloaded = (
- run_cmd(["git", "rev-parse", "--quiet", "--verify", "%s^{commit}" % merge_hash]).strip()
- != ""
+ run_cmd(["git", "rev-parse", "--quiet", "--verify", "%s^{commit}" % merge_hash]).strip() != ""
)
if not commit_is_downloaded:
fail("Couldn't find any merge commit for #%s, you may need to update HEAD." % pr_num)
- print("Found commit %s:\n%s" % (merge_hash, message))
+ print("Found commit {}:\n{}".format(merge_hash, message))
cherry_pick(pr_num, merge_hash, latest_branch)
sys.exit(0)
if not bool(pr["mergeable"]):
- msg = (
- "Pull request %s is not mergeable in its current form.\n" % pr_num
- + "Continue? (experts only!)"
- )
+ msg = "Pull request %s is not mergeable in its current form.\n" % pr_num + "Continue? (experts only!)"
continue_maybe(msg)
print("\n=== Pull Request #%s ===" % pr_num)
-print("title\t%s\nsource\t%s\ntarget\t%s\nurl\t%s" % (title, pr_repo_desc, target_ref, url))
+print("title\t{}\nsource\t{}\ntarget\t{}\nurl\t{}".format(title, pr_repo_desc, target_ref, url))
continue_maybe("Proceed with merging pull request #%s?" % pr_num)
merged_refs = [target_ref]
@@ -370,7 +355,7 @@ while input("\n%s (y/n): " % pick_prompt).lower() == "y":
if JIRA_IMPORTED:
if JIRA_USERNAME and JIRA_PASSWORD:
continue_maybe("Would you like to update an associated JIRA?")
- jira_comment = "Issue resolved by pull request %s\n[%s/%s]" % (pr_num, GITHUB_BASE, pr_num)
+ jira_comment = "Issue resolved by pull request {}\n[{}/{}]".format(pr_num, GITHUB_BASE, pr_num)
resolve_jira_issues(title, merged_refs, jira_comment)
else:
print("JIRA_USERNAME and JIRA_PASSWORD not set")
diff --git a/dev-support/database/init-database.py b/dev-support/database/init-database.py
index 6df1131b..fd0350c2 100644
--- a/dev-support/database/init-database.py
+++ b/dev-support/database/init-database.py
@@ -35,7 +35,7 @@ def commit(sql):
conn.commit()
except mysql.connector.Error as err:
- print("Something went wrong: {}".format(err))
+ print(f"Something went wrong: {err}")
# Rolling back in case of error
conn.rollback()
diff --git a/dev-support/examples/mnist-pytorch/DDP/mnist_distributed.py b/dev-support/examples/mnist-pytorch/DDP/mnist_distributed.py
index 110f314f..c8ac92b6 100644
--- a/dev-support/examples/mnist-pytorch/DDP/mnist_distributed.py
+++ b/dev-support/examples/mnist-pytorch/DDP/mnist_distributed.py
@@ -14,8 +14,8 @@
specific language governing permissions and limitations
under the License.
"""
+# Reference: https://github.com/kubeflow/pytorch-operator/blob/master/examples/mnist/mnist.py
-from __future__ import print_function
import argparse
import os
@@ -33,12 +33,12 @@ import submarine
WORLD_SIZE = int(os.environ.get("WORLD_SIZE", 1))
rank = int(os.environ.get("RANK", 0))
-print("WORLD={} , RANK={}".format(WORLD_SIZE, rank))
+print(f"WORLD={WORLD_SIZE} , RANK={rank}")
class Net(nn.Module):
def __init__(self):
- super(Net, self).__init__()
+ super().__init__()
self.conv1 = nn.Conv2d(1, 20, 5, 1)
self.conv2 = nn.Conv2d(20, 50, 5, 1)
self.fc1 = nn.Linear(4 * 4 * 50, 500)
@@ -92,7 +92,7 @@ def test(args, model, device, test_loader, writer, epoch):
correct += pred.eq(target.view_as(pred)).sum().item()
test_loss /= len(test_loader.dataset)
- print("\naccuracy={:.4f}\n".format(float(correct) / len(test_loader.dataset)))
+ print(f"\naccuracy={float(correct) / len(test_loader.dataset):.4f}\n")
writer.add_scalar("accuracy", float(correct) / len(test_loader.dataset), epoch)
submarine.log_metric("accuracy", float(correct) / len(test_loader.dataset), epoch)
@@ -125,15 +125,11 @@ if __name__ == "__main__":
parser.add_argument(
"--epochs", type=int, default=5, metavar="N", help="number of epochs to train (default: 5)"
)
- parser.add_argument(
- "--lr", type=float, default=0.01, metavar="LR", help="learning rate (default: 0.01)"
- )
+ parser.add_argument("--lr", type=float, default=0.01, metavar="LR", help="learning rate (default: 0.01)")
parser.add_argument(
"--momentum", type=float, default=0.5, metavar="M", help="SGD momentum (default: 0.5)"
)
- parser.add_argument(
- "--no-cuda", action="store_true", default=False, help="disables CUDA training"
- )
+ parser.add_argument("--no-cuda", action="store_true", default=False, help="disables CUDA training")
parser.add_argument("--seed", type=int, default=1, metavar="S", help="random seed (default: 1)")
parser.add_argument(
"--log-interval",
@@ -145,9 +141,7 @@ if __name__ == "__main__":
parser.add_argument(
"--save-model", action="store_true", default=False, help="For Saving the current Model"
)
- parser.add_argument(
- "--dir", default="logs", metavar="L", help="directory where summary logs are stored"
- )
+ parser.add_argument("--dir", default="logs", metavar="L", help="directory where summary logs are stored")
if dist.is_available():
parser.add_argument(
"--backend",
@@ -170,7 +164,7 @@ if __name__ == "__main__":
device = torch.device("cuda" if use_cuda else "cpu")
if should_distribute():
- print("Using distributed PyTorch with {} backend".format(args.backend))
+ print(f"Using distributed PyTorch with {args.backend} backend")
dist.init_process_group(backend=args.backend, world_size=WORLD_SIZE, rank=rank)
kwargs = {"num_workers": 1, "pin_memory": True} if use_cuda else {}
@@ -179,9 +173,7 @@ if __name__ == "__main__":
"../data",
train=True,
download=True,
- transform=transforms.Compose(
- [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
- ),
+ transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]),
)
train_sampler = torch.utils.data.distributed.DistributedSampler(
train_dataset, num_replicas=WORLD_SIZE, rank=rank
@@ -192,29 +184,25 @@ if __name__ == "__main__":
batch_size=args.batch_size,
shuffle=False,
sampler=train_sampler,
- **kwargs
+ **kwargs,
)
test_loader = torch.utils.data.DataLoader(
datasets.FashionMNIST(
"../data",
train=False,
- transform=transforms.Compose(
- [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
- ),
+ transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]),
),
batch_size=args.test_batch_size,
shuffle=False,
- **kwargs
+ **kwargs,
)
model = Net().to(device)
if is_distributed():
Distributor = (
- nn.parallel.DistributedDataParallel
- if use_cuda
- else nn.parallel.DistributedDataParallelCPU
+ nn.parallel.DistributedDataParallel if use_cuda else nn.parallel.DistributedDataParallelCPU
)
model = Distributor(model)
@@ -226,8 +214,3 @@ if __name__ == "__main__":
test(args, model, device, test_loader, writer, epoch)
if args.save_model:
torch.save(model.state_dict(), "mnist_cnn.pt")
-
-"""
-Reference:
-https://github.com/kubeflow/pytorch-operator/blob/master/examples/mnist/mnist.py
-"""
diff --git a/dev-support/examples/mnist-tensorflow/MirroredStrategy/mnist_keras_distributed.py b/dev-support/examples/mnist-tensorflow/MirroredStrategy/mnist_keras_distributed.py
index 943e9e21..35df4981 100644
--- a/dev-support/examples/mnist-tensorflow/MirroredStrategy/mnist_keras_distributed.py
+++ b/dev-support/examples/mnist-tensorflow/MirroredStrategy/mnist_keras_distributed.py
@@ -14,6 +14,7 @@
specific language governing permissions and limitations
under the License.
"""
+# Reference: https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy
import os
@@ -27,7 +28,7 @@ mnist_train, mnist_test = datasets["train"], datasets["test"]
strategy = tf.distribute.MirroredStrategy()
-print("Number of devices: {}".format(strategy.num_replicas_in_sync))
+print(f"Number of devices: {strategy.num_replicas_in_sync}")
# You can also do info.splits.total_num_examples to get the total
# number of examples in the dataset.
@@ -88,7 +89,7 @@ def decay(epoch):
# Define a callback for printing the learning rate at the end of each epoch.
class PrintLR(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
- print("\nLearning rate for epoch {} is {}".format(epoch + 1, model.optimizer.lr.numpy()))
+ print(f"\nLearning rate for epoch {epoch + 1} is {model.optimizer.lr.numpy()}")
submarine.log_metric("lr", model.optimizer.lr.numpy())
submarine.save_model(model, "tensorflow", "mnist-tf")
@@ -109,9 +110,6 @@ if __name__ == "__main__":
submarine.log_metric("Val_accuracy", hist.history["accuracy"][i], i)
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
eval_loss, eval_acc = model.evaluate(eval_dataset)
- print("Eval loss: {}, Eval accuracy: {}".format(eval_loss, eval_acc))
+ print(f"Eval loss: {eval_loss}, Eval accuracy: {eval_acc}")
submarine.log_param("loss", eval_loss)
submarine.log_param("acc", eval_acc)
-"""Reference:
-https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy
-"""
diff --git a/dev-support/examples/mnist-tensorflow/MultiWorkerMirroredStrategy/mnist_keras_distributed.py b/dev-support/examples/mnist-tensorflow/MultiWorkerMirroredStrategy/mnist_keras_distributed.py
index d5353777..791bffc8 100644
--- a/dev-support/examples/mnist-tensorflow/MultiWorkerMirroredStrategy/mnist_keras_distributed.py
+++ b/dev-support/examples/mnist-tensorflow/MultiWorkerMirroredStrategy/mnist_keras_distributed.py
@@ -14,6 +14,10 @@
specific language governing permissions and limitations
under the License.
"""
+# References
+# https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras
+# https://reurl.cc/no9Zk8
+
import json
import os
@@ -98,9 +102,3 @@ if __name__ == "__main__":
for i in range(EPOCHS):
submarine.log_metric("val_loss", hist.history["loss"][i], i)
submarine.log_metric("Val_accuracy", hist.history["accuracy"][i], i)
-
-
-"""Reference
-https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras
-https://reurl.cc/no9Zk8
-"""
diff --git a/dev-support/examples/mnist-tensorflow/ParameterServerStrategy/mnist_keras_distributed.py b/dev-support/examples/mnist-tensorflow/ParameterServerStrategy/mnist_keras_distributed.py
index d1e14dbb..2372ee98 100644
--- a/dev-support/examples/mnist-tensorflow/ParameterServerStrategy/mnist_keras_distributed.py
+++ b/dev-support/examples/mnist-tensorflow/ParameterServerStrategy/mnist_keras_distributed.py
@@ -14,6 +14,8 @@
specific language governing permissions and limitations
under the License.
"""
+# Reference: https://www.tensorflow.org/tutorials/distribute/parameter_server_training
+
import json
import os
@@ -80,8 +82,3 @@ if __name__ == "__main__":
submarine.log_metric("val_loss", hist.history["loss"][i], i)
submarine.log_metric("Val_accuracy", hist.history["accuracy"][i], i)
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
-
-"""
-Reference:
-https://www.tensorflow.org/tutorials/distribute/parameter_server_training
-"""
diff --git a/dev-support/examples/nn-pytorch/model.py b/dev-support/examples/nn-pytorch/model.py
index c591949b..662c08f5 100644
--- a/dev-support/examples/nn-pytorch/model.py
+++ b/dev-support/examples/nn-pytorch/model.py
@@ -21,7 +21,7 @@ import submarine
class LinearNNModel(torch.nn.Module):
def __init__(self):
- super(LinearNNModel, self).__init__()
+ super().__init__()
self.linear = torch.nn.Linear(2, 1) # One in and one out
def forward(self, x):
diff --git a/dev-support/examples/quickstart/train.py b/dev-support/examples/quickstart/train.py
index 51d92cf0..5653fa97 100644
--- a/dev-support/examples/quickstart/train.py
+++ b/dev-support/examples/quickstart/train.py
@@ -71,9 +71,7 @@ def main():
with strategy.scope():
ds_train = make_datasets_unbatched().batch(BATCH_SIZE).repeat()
options = tf.data.Options()
- options.experimental_distribute.auto_shard_policy = (
- tf.data.experimental.AutoShardPolicy.DATA
- )
+ options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA
ds_train = ds_train.with_options(options)
# Model building/compiling need to be within `strategy.scope()`.
multi_worker_model = build_and_compile_cnn_model()
diff --git a/dev-support/mini-submarine/submarine/image_classification.py b/dev-support/mini-submarine/submarine/image_classification.py
index fad373f1..aa3aa2b9 100644
--- a/dev-support/mini-submarine/submarine/image_classification.py
+++ b/dev-support/mini-submarine/submarine/image_classification.py
@@ -13,7 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from __future__ import division
import argparse
import logging
@@ -67,9 +66,7 @@ parser.add_argument(
type=int,
help="number of workers for dataloader",
)
-parser.add_argument(
- "--batch-size", type=int, default=32, help="training batch size per device (CPU/GPU)."
-)
+parser.add_argument("--batch-size", type=int, default=32, help="training batch size per device (CPU/GPU).")
parser.add_argument(
"--gpus",
type=str,
@@ -81,9 +78,7 @@ parser.add_argument("--lr", type=float, default=0.1, help="learning rate. defaul
parser.add_argument(
"--momentum", type=float, default=0.9, help="momentum value for optimizer, default is 0.9."
)
-parser.add_argument(
- "--wd", type=float, default=0.0001, help="weight decay rate. default is 0.0001."
-)
+parser.add_argument("--wd", type=float, default=0.0001, help="weight decay rate. default is 0.0001.")
parser.add_argument("--seed", type=int, default=123, help="random seed to use. Default=123.")
parser.add_argument(
"--mode",
@@ -101,9 +96,7 @@ parser.add_argument(
action="store_true",
help="enable batch normalization or not in vgg. default is false.",
)
-parser.add_argument(
- "--use-pretrained", action="store_true", help="enable using pretrained model from gluon."
-)
+parser.add_argument("--use-pretrained", action="store_true", help="enable using pretrained model from gluon.")
parser.add_argument(
"--prefix",
default="",
@@ -113,9 +106,7 @@ parser.add_argument(
parser.add_argument(
"--start-epoch", default=0, type=int, help="starting epoch, 0 for fresh training, > 0 to resume"
)
-parser.add_argument(
- "--resume", type=str, default="", help="path to saved weight where you want resume"
-)
+parser.add_argument("--resume", type=str, default="", help="path to saved weight where you want resume")
parser.add_argument("--lr-factor", default=0.1, type=float, help="learning rate decay ratio")
parser.add_argument(
"--lr-steps", default="30,60,90", type=str, help="list of learning rate decay epochs as in str"
@@ -129,12 +120,8 @@ parser.add_argument(
type=int,
help="epoch frequence to save model, best model will always be saved",
)
-parser.add_argument(
- "--kvstore", type=str, default="device", help="kvstore to use for trainer/module."
-)
-parser.add_argument(
- "--log-interval", type=int, default=50, help="Number of batches to wait before logging."
-)
+parser.add_argument("--kvstore", type=str, default="device", help="kvstore to use for trainer/module.")
+parser.add_argument("--log-interval", type=int, default=50, help="Number of batches to wait before logging.")
parser.add_argument(
"--profile",
action="store_true",
@@ -143,9 +130,7 @@ parser.add_argument(
"and prints out the memory usage by python function at the end."
),
)
-parser.add_argument(
- "--builtin-profiler", type=int, default=0, help="Enable built-in profiler (0=off, 1=on)"
-)
+parser.add_argument("--builtin-profiler", type=int, default=0, help="Enable built-in profiler (0=off, 1=on)")
opt = parser.parse_args()
# global variables
@@ -192,9 +177,7 @@ def get_cifar10_iterator(batch_size, data_shape, resize=-1, num_parts=1, part_in
def get_imagenet_transforms(data_shape=224, dtype="float32"):
def train_transform(image, label):
- image, _ = mx.image.random_size_crop(
- image, (data_shape, data_shape), 0.08, (3 / 4.0, 4 / 3.0)
- )
+ image, _ = mx.image.random_size_crop(image, (data_shape, data_shape), 0.08, (3 / 4.0, 4 / 3.0))
image = mx.nd.image.random_flip_left_right(image)
image = mx.nd.image.to_tensor(image)
image = mx.nd.image.normalize(image, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
@@ -247,7 +230,7 @@ def get_caltech101_data():
tar.close()
print("Data extracted")
training_path = os.path.join(data_folder, dataset_name)
- testing_path = os.path.join(data_folder, "{}_test".format(dataset_name))
+ testing_path = os.path.join(data_folder, f"{dataset_name}_test")
return training_path, testing_path
@@ -272,7 +255,7 @@ def get_caltech101_iterator(batch_size, num_workers, dtype):
class DummyIter(mx.io.DataIter):
def __init__(self, batch_size, data_shape, batches=100):
- super(DummyIter, self).__init__(batch_size)
+ super().__init__(batch_size)
self.data_shape = (batch_size,) + data_shape
self.label_shape = (batch_size,)
self.provide_data = [("data", self.data_shape)]
@@ -297,13 +280,11 @@ def dummy_iterator(batch_size, data_shape):
class ImagePairIter(mx.io.DataIter):
- def __init__(
- self, path, data_shape, label_shape, batch_size=64, flag=0, input_aug=None, target_aug=None
- ):
+ def __init__(self, path, data_shape, label_shape, batch_size=64, flag=0, input_aug=None, target_aug=None):
def is_image_file(fn):
return any(fn.endswith(ext) for ext in [".png", ".jpg", ".jpeg"])
- super(ImagePairIter, self).__init__(batch_size)
+ super().__init__(batch_size)
self.data_shape = (batch_size,) + data_shape
self.label_shape = (batch_size,) + label_shape
self.input_aug = input_aug
@@ -389,7 +370,7 @@ def get_data_iters(dataset, batch_size, opt):
if not opt.data_dir:
raise ValueError(
"Dir containing raw images in train/val is required for imagenet."
- 'Please specify "--data-dir"'
+ "Please specify \"--data-dir\""
)
train_data, val_data = get_imagenet_iterator(
@@ -466,12 +447,8 @@ def train(opt, ctx):
metric.reset()
btic = time.time()
for i, batch in enumerate(train_data):
- data = gluon.utils.split_and_load(
- batch.data[0].astype(opt.dtype), ctx_list=ctx, batch_axis=0
- )
- label = gluon.utils.split_and_load(
- batch.label[0].astype(opt.dtype), ctx_list=ctx, batch_axis=0
- )
+ data = gluon.utils.split_and_load(batch.data[0].astype(opt.dtype), ctx_list=ctx, batch_axis=0)
+ label = gluon.utils.split_and_load(batch.label[0].astype(opt.dtype), ctx_list=ctx, batch_axis=0)
outputs = []
Ls = []
with ag.record():
@@ -513,15 +490,12 @@ def train(opt, ctx):
logger.info("[Epoch %d] training: %s=%f, %s=%f" % (epoch, name[0], acc[0], name[1], acc[1]))
logger.info("[Epoch %d] time cost: %f" % (epoch, epoch_time))
name, val_acc = test(ctx, val_data)
- logger.info(
- "[Epoch %d] validation: %s=%f, %s=%f"
- % (epoch, name[0], val_acc[0], name[1], val_acc[1])
- )
+ logger.info("[Epoch %d] validation: %s=%f, %s=%f" % (epoch, name[0], val_acc[0], name[1], val_acc[1]))
# save model if meet requirements
save_checkpoint(epoch, val_acc[0], best_acc)
if num_epochs > 1:
- print("Average epoch time: {}".format(float(total_time) / (num_epochs - 1)))
+ print(f"Average epoch time: {float(total_time) / (num_epochs - 1)}")
def main():
@@ -569,10 +543,10 @@ if __name__ == "__main__":
import hotshot
import hotshot.stats
- prof = hotshot.Profile("image-classifier-%s-%s.prof" % (opt.model, opt.mode))
+ prof = hotshot.Profile(f"image-classifier-{opt.model}-{opt.mode}.prof")
prof.runcall(main)
prof.close()
- stats = hotshot.stats.load("image-classifier-%s-%s.prof" % (opt.model, opt.mode))
+ stats = hotshot.stats.load(f"image-classifier-{opt.model}-{opt.mode}.prof")
stats.strip_dirs()
stats.sort_stats("cumtime", "calls")
stats.print_stats()
diff --git a/dev-support/mini-submarine/submarine/mnist_distributed.py b/dev-support/mini-submarine/submarine/mnist_distributed.py
index 8ad3c9e0..0f37fbf8 100644
--- a/dev-support/mini-submarine/submarine/mnist_distributed.py
+++ b/dev-support/mini-submarine/submarine/mnist_distributed.py
@@ -24,7 +24,6 @@ using the Adam optimizer, updating the model parameters on shared parameter serv
The current training accuracy is printed out after every 100 steps.
"""
-from __future__ import absolute_import, division, print_function
import json
import logging
@@ -44,9 +43,7 @@ TB_PORT_ENV_VAR = "TB_PORT"
tf.flags.DEFINE_string("mnist_data_url", "", "Url for mnist handwritten digits dataset")
# Input/output directories
-tf.flags.DEFINE_string(
- "data_dir", "/tmp/tensorflow/mnist/input_data", "Directory for storing input data"
-)
+tf.flags.DEFINE_string("data_dir", "/tmp/tensorflow/mnist/input_data", "Directory for storing input data")
tf.flags.DEFINE_string(
"working_dir",
"/tmp/tensorflow/mnist/working_dir",
@@ -206,9 +203,7 @@ def main(_):
elif job_name == "worker":
# Create our model graph. Assigns ops to the local worker by default.
with tf.device(
- tf.train.replica_device_setter(
- worker_device="/job:worker/task:%d" % task_index, cluster=cluster
- )
+ tf.train.replica_device_setter(worker_device="/job:worker/task:%d" % task_index, cluster=cluster)
):
features, labels, keep_prob, global_step, train_step, accuracy, merged = create_model()
@@ -223,9 +218,7 @@ def main(_):
# avoid hanging issues when one worker finishes. We are using
# asynchronous training so there is no need for the workers to
# communicate.
- config_proto = tf.ConfigProto(
- device_filters=["/job:ps", "/job:worker/task:%d" % task_index]
- )
+ config_proto = tf.ConfigProto(device_filters=["/job:ps", "/job:worker/task:%d" % task_index])
with tf.train.MonitoredTrainingSession(
master=server.target,
diff --git a/dev-support/mini-submarine/submarine/pytorch_mnist_distributed.py b/dev-support/mini-submarine/submarine/pytorch_mnist_distributed.py
index 63c1ecf1..7d001354 100644
--- a/dev-support/mini-submarine/submarine/pytorch_mnist_distributed.py
+++ b/dev-support/mini-submarine/submarine/pytorch_mnist_distributed.py
@@ -24,7 +24,6 @@ using the Adam optimizer, updating the model parameters on shared parameter serv
The current training accuracy is printed out after every 100 steps.
"""
-from __future__ import division, print_function
import argparse
import os
@@ -37,7 +36,7 @@ from torch.utils.data.distributed import DistributedSampler
from torchvision import datasets, transforms
-class AverageMeter(object):
+class AverageMeter:
def __init__(self):
self.sum = 0
self.count = 0
@@ -51,7 +50,7 @@ class AverageMeter(object):
return self.sum / self.count
-class AccuracyMeter(object):
+class AccuracyMeter:
def __init__(self):
self.correct = 0
self.count = 0
@@ -68,7 +67,7 @@ class AccuracyMeter(object):
return self.correct / self.count
-class Trainer(object):
+class Trainer:
def __init__(self, net, optimizer, train_loader, test_loader, device):
self.net = net
self.optimizer = optimizer
@@ -136,7 +135,7 @@ class Trainer(object):
class Net(nn.Module):
def __init__(self):
- super(Net, self).__init__()
+ super().__init__()
self.fc = nn.Linear(784, 10)
def forward(self, x):
@@ -155,9 +154,7 @@ def get_dataloader(root, batch_size):
train_set = datasets.MNIST(root, train=True, transform=transform, download=True)
sampler = DistributedSampler(train_set)
- train_loader = DataLoader(
- train_set, batch_size=batch_size, shuffle=(sampler is None), sampler=sampler
- )
+ train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=(sampler is None), sampler=sampler)
test_loader = DataLoader(
datasets.MNIST(root, train=False, transform=transform, download=True),
@@ -184,7 +181,7 @@ def solve(args):
test_loss, test_acc = trainer.evaluate()
print(
- "Epoch: {}/{},".format(epoch, args.epochs),
+ f"Epoch: {epoch}/{args.epochs},",
"train loss: {:.6f}, train acc: {:.6f}, test loss: {:.6f}, test acc: {:.6f}.".format(
train_loss, train_acc, test_loss, test_acc
),
diff --git a/dev-support/style-check/python/auto-format.sh b/dev-support/style-check/python/auto-format.sh
index f8426916..f9191482 100755
--- a/dev-support/style-check/python/auto-format.sh
+++ b/dev-support/style-check/python/auto-format.sh
@@ -23,6 +23,6 @@ cd ../../../
# Sort imports
isort submarine-sdk/ dev-support/ website/
# Autoformat code
-black submarine-sdk/ dev-support/ website/ --experimental-string-processing
+black submarine-sdk/ dev-support/ website/
-set +euxo pipefail
\ No newline at end of file
+set +euxo pipefail
diff --git a/dev-support/style-check/python/lint-requirements.txt b/dev-support/style-check/python/lint-requirements.txt
index 798b70f5..3e82ae86 100644
--- a/dev-support/style-check/python/lint-requirements.txt
+++ b/dev-support/style-check/python/lint-requirements.txt
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-click==8.0.4 # For black to work properly. See SUBMARINE-1249.
-black[jupyter]==21.8b0
-flake8==3.9.2
-isort==5.9.3
+black[jupyter]==22.8.0
+click==8.1.3
+flake8==5.0.4
+isort==5.10.1
diff --git a/dev-support/style-check/python/lint.sh b/dev-support/style-check/python/lint.sh
index 596e6c5e..1f40b09b 100755
--- a/dev-support/style-check/python/lint.sh
+++ b/dev-support/style-check/python/lint.sh
@@ -23,10 +23,10 @@ cd ../../../
# Check imports
isort -c submarine-sdk/ dev-support/ website/
# Check code format
-black submarine-sdk/ dev-support/ website/ --check --experimental-string-processing
+black submarine-sdk/ dev-support/ website/docs --check --preview --diff --color
# Check lint: Not checking the code in website
flake8 submarine-sdk/ dev-support/
# Check static typing for Python
-mypy --ignore-missing-imports submarine-sdk/pysubmarine/submarine/
+mypy submarine-sdk/pysubmarine/submarine/
set +euxo pipefail
diff --git a/dev-support/style-check/python/mypy-requirements.txt b/dev-support/style-check/python/mypy-requirements.txt
index 41d5e73c..53f32ba7 100644
--- a/dev-support/style-check/python/mypy-requirements.txt
+++ b/dev-support/style-check/python/mypy-requirements.txt
@@ -13,11 +13,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-mypy==0.930
-types-requests==2.25.6
-types-certifi==2020.4.0
-types-six==1.16.1
-types-python-dateutil==2.8.0
-types-dataclasses==0.6.1
-types-PyYAML==6.0.1
+mypy
sqlalchemy[mypy]
+types-certifi
+types-python-dateutil
+types-PyYAML
+types-requests
diff --git a/dev-support/submarine-installer/scripts/combine-docker-daemons.py b/dev-support/submarine-installer/scripts/combine-docker-daemons.py
index 60da4b8e..f2acceb1 100644
--- a/dev-support/submarine-installer/scripts/combine-docker-daemons.py
+++ b/dev-support/submarine-installer/scripts/combine-docker-daemons.py
@@ -31,6 +31,6 @@ def combineJsons(jsonFile1, jsonFile2, outputFile):
if __name__ == "__main__":
if len(sys.argv) < 4:
- raise Exception(u"3 arguments needed")
+ raise Exception("3 arguments needed")
print(combineJsons(sys.argv[1], sys.argv[2], sys.argv[3]))
diff --git a/pyproject.toml b/pyproject.toml
index cdf4ebd1..26c7bd76 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -17,9 +17,16 @@
[tool.isort]
profile = "black"
-line_length = 100
+
[tool.black]
-max-line-length = 100
-line-length = 100
+line-length = 110
+target-version = ["py37", "py38", "py39", "py310"]
+skip-string-normalization = true
+
[tool.mypy]
-plugins = "sqlalchemy.ext.mypy.plugin"
\ No newline at end of file
+plugins = "sqlalchemy.ext.mypy.plugin"
+ignore_missing_imports = true # suppress `Cannot find implementation or library stub` error
+
+[tool.pytest.ini_options]
+addopts = "--strict-markers"
+markers = ["e2e"]
diff --git a/submarine-experiment-prehandler/fs_prehandler/__init__.py b/submarine-experiment-prehandler/fs_prehandler/__init__.py
index f4540561..363f5df2 100644
--- a/submarine-experiment-prehandler/fs_prehandler/__init__.py
+++ b/submarine-experiment-prehandler/fs_prehandler/__init__.py
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from fs_prehandler.fs_prehandler import FsPreHandler
+from fs_prehandler.fs_prehandler import FsPreHandler # noqa: F401
diff --git a/submarine-experiment-prehandler/fs_prehandler/fs_prehandler.py b/submarine-experiment-prehandler/fs_prehandler/fs_prehandler.py
index 0a812181..8af33ff7 100644
--- a/submarine-experiment-prehandler/fs_prehandler/fs_prehandler.py
+++ b/submarine-experiment-prehandler/fs_prehandler/fs_prehandler.py
@@ -15,6 +15,7 @@
from abc import ABC, abstractmethod
+
class FsPreHandler(ABC):
@abstractmethod
def __init__(self):
diff --git a/submarine-experiment-prehandler/fs_prehandler/hdfs_prehandler.py b/submarine-experiment-prehandler/fs_prehandler/hdfs_prehandler.py
index 60265d3c..1a13f1b2 100644
--- a/submarine-experiment-prehandler/fs_prehandler/hdfs_prehandler.py
+++ b/submarine-experiment-prehandler/fs_prehandler/hdfs_prehandler.py
@@ -19,13 +19,14 @@ import os
from fs_prehandler import FsPreHandler
from fsspec.implementations.arrow import HadoopFileSystem
+
class HDFSPreHandler(FsPreHandler):
def __init__(self):
- self.hdfs_host=os.environ['HDFS_HOST']
- self.hdfs_port=int(os.environ['HDFS_PORT'])
- self.hdfs_source=os.environ['HDFS_SOURCE']
- self.dest_path=os.environ['DEST_PATH']
- self.enable_kerberos=os.environ['ENABLE_KERBEROS']
+ self.hdfs_host = os.environ['HDFS_HOST']
+ self.hdfs_port = int(os.environ['HDFS_PORT'])
+ self.hdfs_source = os.environ['HDFS_SOURCE']
+ self.dest_path = os.environ['DEST_PATH']
+ self.enable_kerberos = os.environ['ENABLE_KERBEROS']
logging.info('HDFS_HOST:%s' % self.hdfs_host)
logging.info('HDFS_PORT:%d' % self.hdfs_port)
@@ -37,4 +38,7 @@ class HDFSPreHandler(FsPreHandler):
def process(self):
self.fs.get(self.hdfs_source, self.dest_path, recursive=True)
- logging.info('fetch data from hdfs://%s:%d/%s to %s complete' % (self.hdfs_host, self.hdfs_port, self.hdfs_source, self.dest_path))
+ logging.info(
+ 'fetch data from hdfs://%s:%d/%s to %s complete'
+ % (self.hdfs_host, self.hdfs_port, self.hdfs_source, self.dest_path)
+ )
diff --git a/submarine-experiment-prehandler/prehandler_main.py b/submarine-experiment-prehandler/prehandler_main.py
index 7e14a926..7c5576f8 100644
--- a/submarine-experiment-prehandler/prehandler_main.py
+++ b/submarine-experiment-prehandler/prehandler_main.py
@@ -13,14 +13,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
-from fs_prehandler.hdfs_prehandler import HDFSPreHandler
+
+from fs_prehandler.hdfs_prehandler import HDFSPreHandler
+
def get_file_system_prehandler():
if os.environ['FILE_SYSTEM_TYPE'] == 'HDFS':
return HDFSPreHandler()
+
if __name__ == "__main__":
fsPrehandler = get_file_system_prehandler()
fsPrehandler.process()
-
-
\ No newline at end of file
diff --git a/submarine-sdk/pysubmarine/example/pytorch/afm/run_afm.py b/submarine-sdk/pysubmarine/example/pytorch/afm/run_afm.py
index c086c8c6..c9bee7c9 100644
--- a/submarine-sdk/pysubmarine/example/pytorch/afm/run_afm.py
+++ b/submarine-sdk/pysubmarine/example/pytorch/afm/run_afm.py
@@ -21,9 +21,7 @@ from submarine.ml.pytorch.model.ctr import AFM
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-conf", help="a JSON configuration file for AFM", type=str)
- parser.add_argument(
- "-task_type", default="train", help="train or evaluate, by default is train"
- )
+ parser.add_argument("-task_type", default="train", help="train or evaluate, by default is train")
args = parser.parse_args()
trainer = AFM(json_path=args.conf)
diff --git a/submarine-sdk/pysubmarine/example/pytorch/deepfm/run_deepfm.py b/submarine-sdk/pysubmarine/example/pytorch/deepfm/run_deepfm.py
index 6d3d6223..665752f0 100644
--- a/submarine-sdk/pysubmarine/example/pytorch/deepfm/run_deepfm.py
+++ b/submarine-sdk/pysubmarine/example/pytorch/deepfm/run_deepfm.py
@@ -21,9 +21,7 @@ from submarine.ml.pytorch.model.ctr import DeepFM
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-conf", help="a JSON configuration file for DeepFM", type=str)
- parser.add_argument(
- "-task_type", default="train", help="train or evaluate, by default is train"
- )
+ parser.add_argument("-task_type", default="train", help="train or evaluate, by default is train")
args = parser.parse_args()
trainer = DeepFM(json_path=args.conf)
diff --git a/submarine-sdk/pysubmarine/example/tensorflow/ccpm/run_ccpm.py b/submarine-sdk/pysubmarine/example/tensorflow/ccpm/run_ccpm.py
index 1d21bbcb..71a7fd5d 100644
--- a/submarine-sdk/pysubmarine/example/tensorflow/ccpm/run_ccpm.py
+++ b/submarine-sdk/pysubmarine/example/tensorflow/ccpm/run_ccpm.py
@@ -20,9 +20,7 @@ from submarine.ml.tensorflow.model import CCPM
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-conf", help="a JSON configuration file for CCPM", type=str)
- parser.add_argument(
- "-task_type", default="train", help="train or evaluate, by default is train"
- )
+ parser.add_argument("-task_type", default="train", help="train or evaluate, by default is train")
args = parser.parse_args()
json_path = args.conf
task_type = args.task_type
diff --git a/submarine-sdk/pysubmarine/example/tensorflow/deepfm/run_deepfm.py b/submarine-sdk/pysubmarine/example/tensorflow/deepfm/run_deepfm.py
index d1c4df65..56d01ecc 100644
--- a/submarine-sdk/pysubmarine/example/tensorflow/deepfm/run_deepfm.py
+++ b/submarine-sdk/pysubmarine/example/tensorflow/deepfm/run_deepfm.py
@@ -20,9 +20,7 @@ from submarine.ml.tensorflow.model import DeepFM
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-conf", help="a JSON configuration file for DeepFM", type=str)
- parser.add_argument(
- "-task_type", default="train", help="train or evaluate, by default is train"
- )
+ parser.add_argument("-task_type", default="train", help="train or evaluate, by default is train")
args = parser.parse_args()
json_path = args.conf
task_type = args.task_type
diff --git a/submarine-sdk/pysubmarine/example/tensorflow/fm/run_fm.py b/submarine-sdk/pysubmarine/example/tensorflow/fm/run_fm.py
index a4f447cc..16b95a2a 100644
--- a/submarine-sdk/pysubmarine/example/tensorflow/fm/run_fm.py
+++ b/submarine-sdk/pysubmarine/example/tensorflow/fm/run_fm.py
@@ -20,9 +20,7 @@ from submarine.ml.tensorflow.model import FM
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-conf", help="a JSON configuration file for FM", type=str)
- parser.add_argument(
- "-task_type", default="train", help="train or evaluate, by default is train"
- )
+ parser.add_argument("-task_type", default="train", help="train or evaluate, by default is train")
args = parser.parse_args()
json_path = args.conf
task_type = args.task_type
diff --git a/submarine-sdk/pysubmarine/example/tracking.py b/submarine-sdk/pysubmarine/example/tracking.py
index b5d7400a..678e5986 100644
--- a/submarine-sdk/pysubmarine/example/tracking.py
+++ b/submarine-sdk/pysubmarine/example/tracking.py
@@ -25,5 +25,5 @@ if __name__ == "__main__":
submarine.log_param("max_iter", 100)
lr.fit(X, y)
score = lr.score(X, y)
- print("Score: %s" % score)
+ print(f"Score: {score}")
submarine.log_metric("score", score)
diff --git a/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
index 65213655..a8e85c18 100644
--- a/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
+++ b/submarine-sdk/pysubmarine/github-actions/pypi-requirements.txt
@@ -15,5 +15,5 @@
pip==20.1.1
setuptools==47.3.1
-wheel==0.34.2
twine==3.1.1
+wheel==0.34.2
diff --git a/submarine-sdk/pysubmarine/github-actions/test-requirements.txt b/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
index 6c67af3a..1f38d655 100644
--- a/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+++ b/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
@@ -13,21 +13,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-mock==2.0.0
-moto==1.3.7
-pandas<=0.23.4
-scikit-learn==0.20.2
-scipy==1.2.1
-attrdict==2.0.0
-pytest==3.2.1
-pytest-cov==2.6.0
-pytest-localserver==0.5.0
-pylint==2.12.2
-PyMySQL==0.9.3
-pytest-mock==1.13.0
-certifi >= 14.05.14
-six >= 1.10
-python_dateutil >= 2.5.3
-setuptools >= 21.0.0
-urllib3 >= 1.15.1
-freezegun==1.1.0
+certifi>=14.05.14
+freezegun==1.2.2
+pandas>=1.2.0,<=1.3.5 # 1.4.x does not support cp37
+pylint==2.15.2
+PyMySQL==1.0.2
+pytest==7.1.3
+pytest-cov==3.0.0
+pytest-localserver==0.7.0
+python_dateutil>=2.5.3
+scikit-learn>=0.24.2,<=1.0.2 # 1.1.x doest not support cp37
+setuptools>=21.0.0
+urllib3>=1.15.1
diff --git a/submarine-sdk/pysubmarine/setup.py b/submarine-sdk/pysubmarine/setup.py
index b8e217b0..f30c2000 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -28,41 +28,41 @@ setup(
packages=find_packages(exclude=["tests", "tests.*"]),
package_data={"submarine.cli.config": ["cli_config.yaml"]},
install_requires=[
- "six>=1.10.0",
- "numpy==1.19.2",
+ "numpy",
"pandas",
"sqlalchemy>=1.4.0",
"sqlparse",
"pymysql",
- "requests==2.26.0",
+ "requests",
"urllib3>=1.15.1",
"certifi>=14.05.14",
"python-dateutil>=2.5.3",
"pyarrow>=6.0.1",
"boto3>=1.17.58",
- "click==8.0.3",
- "rich==10.15.2",
- "dacite==1.6.0",
- "dataclasses>=0.6",
- "pyaml==21.10.1",
+ "click>=8.1.0",
+ "rich",
+ "dacite",
+ "pyaml",
],
extras_require={
- "tf": ["tensorflow==1.15.0", "protobuf<=3.20.1"],
+ "tf": ["tensorflow==1.15.5", "numpy>=1.16.0,<1.19.0", "protobuf>=3.6.1,<3.20"],
"tf2": [
- "tensorflow==2.6.0",
+ "tensorflow>=2.6.0,<2.10.0",
+ "numpy>=1.14.5",
+ "keras>=2.6.0",
+ "protobuf>=3.9.2,<3.20",
+ "tensorflow-addons==0.17.0",
+ "tensorflow-estimator>=2.9.0rc0,<2.10.0",
"tf_slim==1.1.0",
- "tensorflow-addons==0.14.0",
- "tensorflow-estimator==2.6.0",
- "keras==2.6",
- "protobuf<=3.20.1",
],
"pytorch": ["torch>=1.5.0", "torchvision>=0.6.0"],
},
classifiers=[
"Intended Audience :: Developers",
- "Programming Language :: Python :: 3.5",
- "Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
+ "Programming Language :: Python :: 3.8",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
],
entry_points={
"console_scripts": [
diff --git a/submarine-sdk/pysubmarine/submarine/artifacts/repository.py b/submarine-sdk/pysubmarine/submarine/artifacts/repository.py
index 50e469cb..504727ba 100644
--- a/submarine-sdk/pysubmarine/submarine/artifacts/repository.py
+++ b/submarine-sdk/pysubmarine/submarine/artifacts/repository.py
@@ -49,7 +49,7 @@ class Repository:
def log_artifacts(self, dest_path: str, local_dir: str) -> str:
local_dir = os.path.abspath(local_dir)
- for (root, _, filenames) in os.walk(local_dir):
+ for root, _, filenames in os.walk(local_dir):
upload_path = dest_path
if root != local_dir:
rel_path = os.path.relpath(root, local_dir)
diff --git a/submarine-sdk/pysubmarine/submarine/cli/__init__.py b/submarine-sdk/pysubmarine/submarine/cli/__init__.py
index 8f00b071..73c675d0 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/__init__.py
@@ -26,6 +26,5 @@
Generated by: https://openapi-generator.tech
"""
-from __future__ import absolute_import
__version__ = "0.8.0-SNAPSHOT"
diff --git a/submarine-sdk/pysubmarine/submarine/cli/config/__init__.py b/submarine-sdk/pysubmarine/submarine/cli/config/__init__.py
index 4103e81a..18cc80ab 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/config/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/config/__init__.py
@@ -13,7 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from submarine.cli.config.command import get_config, init_config, list_config, set_config
+from submarine.cli.config.command import (
+ get_config,
+ init_config,
+ list_config,
+ set_config,
+)
__all__ = [
"list_config",
diff --git a/submarine-sdk/pysubmarine/submarine/cli/config/command.py b/submarine-sdk/pysubmarine/submarine/cli/config/command.py
index 8927bc66..32208799 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/config/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/config/command.py
@@ -23,7 +23,13 @@ from rich.console import Console
from rich.json import JSON as richJSON
from rich.panel import Panel
-from submarine.cli.config.config import initConfig, loadConfig, rgetattr, rsetattr, saveConfig
+from submarine.cli.config.config import (
+ initConfig,
+ loadConfig,
+ rgetattr,
+ rsetattr,
+ saveConfig,
+)
@click.command("list")
@@ -41,7 +47,7 @@ def get_config(param):
"""Get Submarine CLI Config"""
_config = loadConfig()
try:
- click.echo("{}={}".format(param, rgetattr(_config, param)))
+ click.echo(f"{param}={rgetattr(_config, param)}")
except AttributeError as err:
click.echo(err)
diff --git a/submarine-sdk/pysubmarine/submarine/cli/config/config.py b/submarine-sdk/pysubmarine/submarine/cli/config/config.py
index 056961b6..085d67fe 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/config/config.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/config/config.py
@@ -38,15 +38,13 @@ class BaseConfig:
if hasattr(_field.type, "__origin__") and _field.type.__origin__ == Union:
if not isinstance(__value, _field.type.__args__):
msg = (
- "Field `{0.name}` is of type {1}, should be one of the type: {0.type.__args__}"
- .format(_field, type(__value))
+ f"Field `{_field.name}` is of type {type(__value)}, should be one of the type:"
+ f" {_field.type.__args__}"
)
raise TypeError(msg)
else:
if not type(__value) == _field.type:
- msg = "Field {0.name} is of type {1}, should be {0.type}".format(
- _field, type(__value)
- )
+ msg = f"Field {_field.name} is of type {type(__value)}, should be {_field.type}"
raise TypeError(msg)
super().__setattr__(__name, __value)
@@ -101,7 +99,7 @@ def rsetattr(obj, attr, val):
def loadConfig(config_path: str = CONFIG_YAML_PATH) -> SubmarineCliConfig:
- with open(config_path, "r") as stream:
+ with open(config_path) as stream:
try:
parsed_yaml: dict = yaml.safe_load(stream)
return_config: SubmarineCliConfig = dacite.from_dict(
diff --git a/submarine-sdk/pysubmarine/submarine/cli/environment/__init__.py b/submarine-sdk/pysubmarine/submarine/cli/environment/__init__.py
index 38a7a274..ccc2d092 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/environment/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/environment/__init__.py
@@ -13,7 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from submarine.cli.environment.command import delete_environment, get_environment, list_environment
+from submarine.cli.environment.command import (
+ delete_environment,
+ get_environment,
+ list_environment,
+)
__all__ = [
"get_environment",
diff --git a/submarine-sdk/pysubmarine/submarine/cli/environment/command.py b/submarine-sdk/pysubmarine/submarine/cli/environment/command.py
index 5b0ace90..0b910e35 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/environment/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/environment/command.py
@@ -31,9 +31,7 @@ from submarine.client.exceptions import ApiException
submarineCliConfig = loadConfig()
environmentClient = EnvironmentClient(
- host="http://{}:{}".format(
- submarineCliConfig.connection.hostname, submarineCliConfig.connection.port
- )
+ host=f"http://{submarineCliConfig.connection.hostname}:{submarineCliConfig.connection.port}"
)
POLLING_INTERVAL = 1 # sec
@@ -81,9 +79,9 @@ def list_environment():
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("environment")
@@ -94,7 +92,7 @@ def get_environment(name):
try:
thread = environmentClient.get_environment_async(name)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Fetching Environment(name = {} )...".format(name)):
+ with console.status(f"[bold green] Fetching Environment(name = {name} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -105,13 +103,13 @@ def get_environment(name):
result = result.result
json_data = richJSON.from_data(result)
- console.print(Panel(json_data, title="Environment(name = {} )".format(name)))
+ console.print(Panel(json_data, title=f"Environment(name = {name} )"))
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("environment")
@@ -122,7 +120,7 @@ def delete_environment(name):
try:
thread = environmentClient.delete_environment_async(name)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Deleting Environment(name = {} )...".format(name)):
+ with console.status(f"[bold green] Deleting Environment(name = {name} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -132,11 +130,11 @@ def delete_environment(name):
result = thread.get()
result = result.result
- console.print("[bold green] Environment(name = {} ) deleted".format(name))
+ console.print(f"[bold green] Environment(name = {name} ) deleted")
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
diff --git a/submarine-sdk/pysubmarine/submarine/cli/experiment/__init__.py b/submarine-sdk/pysubmarine/submarine/cli/experiment/__init__.py
index 1e6253c8..ad4e143d 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/experiment/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/experiment/__init__.py
@@ -13,7 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from submarine.cli.experiment.command import delete_experiment, get_experiment, list_experiment
+from submarine.cli.experiment.command import (
+ delete_experiment,
+ get_experiment,
+ list_experiment,
+)
__all__ = [
"get_experiment",
diff --git a/submarine-sdk/pysubmarine/submarine/cli/experiment/command.py b/submarine-sdk/pysubmarine/submarine/cli/experiment/command.py
index 277c5a0c..78e42d0c 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/experiment/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/experiment/command.py
@@ -31,9 +31,7 @@ from submarine.client.exceptions import ApiException
submarineCliConfig = loadConfig()
experimentClient = ExperimentClient(
- host="http://{}:{}".format(
- submarineCliConfig.connection.hostname, submarineCliConfig.connection.port
- )
+ host=f"http://{submarineCliConfig.connection.hostname}:{submarineCliConfig.connection.port}"
)
POLLING_INTERVAL = 1 # sec
@@ -85,9 +83,9 @@ def list_experiment():
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("experiment")
@@ -98,7 +96,7 @@ def get_experiment(id):
try:
thread = experimentClient.get_experiment_async(id)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Fetching Experiment(id = {} )...".format(id)):
+ with console.status(f"[bold green] Fetching Experiment(id = {id} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -109,13 +107,13 @@ def get_experiment(id):
result = result.result
json_data = richJSON.from_data(result)
- console.print(Panel(json_data, title="Experiment(id = {} )".format(id)))
+ console.print(Panel(json_data, title=f"Experiment(id = {id} )"))
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("experiment")
@@ -127,7 +125,7 @@ def delete_experiment(id, wait):
try:
thread = experimentClient.delete_experiment_async(id)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Deleting Experiment(id = {} )...".format(id)):
+ with console.status(f"[bold green] Deleting Experiment(id = {id} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -139,15 +137,15 @@ def delete_experiment(id, wait):
if wait:
if result["status"] == "Deleted":
- console.print("[bold green] Experiment(id = {} ) deleted".format(id))
+ console.print(f"[bold green] Experiment(id = {id} ) deleted")
else:
console.print("[bold red] Failed")
json_data = richJSON.from_data(result)
- console.print(Panel(json_data, title="Experiment(id = {} )".format(id)))
+ console.print(Panel(json_data, title=f"Experiment(id = {id} )"))
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
diff --git a/submarine-sdk/pysubmarine/submarine/cli/notebook/command.py b/submarine-sdk/pysubmarine/submarine/cli/notebook/command.py
index c6a1d1a4..58248112 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/notebook/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/notebook/command.py
@@ -31,9 +31,7 @@ from submarine.client.exceptions import ApiException
submarineCliConfig = loadConfig()
notebookClient = NotebookClient(
- host="http://{}:{}".format(
- submarineCliConfig.connection.hostname, submarineCliConfig.connection.port
- )
+ host=f"http://{submarineCliConfig.connection.hostname}:{submarineCliConfig.connection.port}"
)
POLLING_INTERVAL = 1 # sec
@@ -85,9 +83,9 @@ def list_notebook():
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("notebook")
@@ -98,7 +96,7 @@ def get_notebook(id):
try:
thread = notebookClient.get_notebook_async(id)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Fetching Notebook(id = {} )...".format(id)):
+ with console.status(f"[bold green] Fetching Notebook(id = {id} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -109,13 +107,13 @@ def get_notebook(id):
result = result.result
json_data = richJSON.from_data(result)
- console.print(Panel(json_data, title="Notebook(id = {} )".format(id)))
+ console.print(Panel(json_data, title=f"Notebook(id = {id} )"))
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
@click.command("notebook")
@@ -126,7 +124,7 @@ def delete_notebook(id):
try:
thread = notebookClient.delete_notebook_async(id)
timeout = time.time() + TIMEOUT
- with console.status("[bold green] Deleting Notebook(id = {} )...".format(id)):
+ with console.status(f"[bold green] Deleting Notebook(id = {id} )..."):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -136,11 +134,11 @@ def delete_notebook(id):
result = thread.get()
result = result.result
- console.print("[bold green] Notebook(id = {} ) deleted".format(id))
+ console.print(f"[bold green] Notebook(id = {id} ) deleted")
except ApiException as err:
if err.body is not None:
errbody = json.loads(err.body)
- click.echo("[Api Error] {}".format(errbody["message"]))
+ click.echo(f"[Api Error] {errbody['message']}")
else:
- click.echo("[Api Error] {}".format(err))
+ click.echo(f"[Api Error] {err}")
diff --git a/submarine-sdk/pysubmarine/submarine/cli/sandbox/command.py b/submarine-sdk/pysubmarine/submarine/cli/sandbox/command.py
index fecaf1a6..52251eab 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/sandbox/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/sandbox/command.py
@@ -22,7 +22,7 @@ import click
@click.option("-v", "--version", "version", help="Specify sandbox version", default="0.6.0")
def start_sandbox(version):
"""Start sandbox"""
- click.echo("start sandbox! version={}".format(version))
+ click.echo(f"start sandbox! version={version}")
@click.command("delete")
diff --git a/submarine-sdk/pysubmarine/submarine/cli/serve/__init__.py b/submarine-sdk/pysubmarine/submarine/cli/serve/__init__.py
index 95edd662..bc24aba4 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/serve/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/serve/__init__.py
@@ -15,7 +15,12 @@
under the License.
"""
-from submarine.cli.serve.command import create_serve, delete_serve, get_serve, list_serve
+from submarine.cli.serve.command import (
+ create_serve,
+ delete_serve,
+ get_serve,
+ list_serve,
+)
__all__ = [
"list_serve",
diff --git a/submarine-sdk/pysubmarine/submarine/cli/serve/command.py b/submarine-sdk/pysubmarine/submarine/cli/serve/command.py
index 066e9495..24441e2a 100644
--- a/submarine-sdk/pysubmarine/submarine/cli/serve/command.py
+++ b/submarine-sdk/pysubmarine/submarine/cli/serve/command.py
@@ -62,9 +62,7 @@ def create_serve(model_name: str, model_version: int):
try:
thread = serveClient.create_serve(model_name, model_version, async_req=True)
timeout = time.time() + TIMEOUT
- with console.status(
- f"[bold green] Creating Serve with name: {model_name}, version: {model_version}"
- ):
+ with console.status(f"[bold green] Creating Serve with name: {model_name}, version: {model_version}"):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -91,9 +89,7 @@ def delete_serve(model_name: str, model_version: int, wait: bool):
try:
thread = serveClient.delete_serve(model_name, model_version, async_req=True)
timeout = time.time() + TIMEOUT
- with console.status(
- f"[bold green] Deleting Serve with name: {model_name}, version: {model_version}"
- ):
+ with console.status(f"[bold green] Deleting Serve with name: {model_name}, version: {model_version}"):
while not thread.ready():
time.sleep(POLLING_INTERVAL)
if time.time() > timeout:
@@ -105,9 +101,7 @@ def delete_serve(model_name: str, model_version: int, wait: bool):
if wait:
if result["status"] == "Deleted":
- console.print(
- f"[bold green] Serve: model name:{model_name}, version: {model_version}"
- )
+ console.print(f"[bold green] Serve: model name:{model_name}, version: {model_version}")
else:
console.print("[bold red] Failed")
json_data = richJSON.from_data(result)
@@ -115,8 +109,7 @@ def delete_serve(model_name: str, model_version: int, wait: bool):
Panel(
json_data,
title=(
- f"[bold green] Serve Deleted: model name:{model_name}, version:"
- f" {model_version}"
+ f"[bold green] Serve Deleted: model name:{model_name}, version: {model_version}"
),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/__init__.py b/submarine-sdk/pysubmarine/submarine/client/__init__.py
index 7860eb3c..c3c38db3 100644
--- a/submarine-sdk/pysubmarine/submarine/client/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/client/__init__.py
@@ -28,8 +28,6 @@
"""
-from __future__ import absolute_import
-
__version__ = "0.8.0-SNAPSHOT"
# import apis into sdk package
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/__init__.py b/submarine-sdk/pysubmarine/submarine/client/api/__init__.py
index 741c497e..eb97a10c 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/__init__.py
@@ -13,7 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from __future__ import absolute_import
# import apis into api package
from submarine.client.api.environment_api import EnvironmentApi
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/environment_api.py b/submarine-sdk/pysubmarine/submarine/client/api/environment_api.py
index 32efa87d..39301cdb 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/environment_api.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/environment_api.py
@@ -24,20 +24,11 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
-from __future__ import absolute_import
-
-import re # noqa: F401
-
-# python 2 and python 3 compatibility library
-import six
-
from submarine.client.api_client import ApiClient
-from submarine.client.exceptions import ApiTypeError, ApiValueError # noqa: F401
+from submarine.client.exceptions import ApiTypeError, ApiValueError
-class EnvironmentApi(object):
+class EnvironmentApi:
"""NOTE: This class is auto generated by OpenAPI Generator
Ref: https://openapi-generator.tech
@@ -100,15 +91,11 @@ class EnvironmentApi(object):
local_var_params = locals()
all_params = ["environment_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method create_environment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method create_environment")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -208,15 +195,11 @@ class EnvironmentApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method delete_environment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method delete_environment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -318,15 +301,11 @@ class EnvironmentApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method get_environment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method get_environment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -428,15 +407,11 @@ class EnvironmentApi(object):
local_var_params = locals()
all_params = ["status"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method list_environment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method list_environment")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -533,15 +508,11 @@ class EnvironmentApi(object):
local_var_params = locals()
all_params = ["id", "environment_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method update_environment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method update_environment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/environment_client.py b/submarine-sdk/pysubmarine/submarine/client/api/environment_client.py
index 3a12dff3..c0de0b0d 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/environment_client.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/environment_client.py
@@ -64,9 +64,7 @@ class EnvironmentClient:
:param environment_spec: submarine environment spec
:return: thread
"""
- thread = self.environment_api.create_environment(
- environment_spec=environment_spec, async_req=True
- )
+ thread = self.environment_api.create_environment(environment_spec=environment_spec, async_req=True)
return thread
def update_environment(self, name, environment_spec):
@@ -76,9 +74,7 @@ class EnvironmentClient:
:param environment_spec: submarine environment spec
:return: submarine environment
"""
- response = self.environment_api.update_environment(
- id=name, environment_spec=environment_spec
- )
+ response = self.environment_api.update_environment(id=name, environment_spec=environment_spec)
return response.result
def get_environment(self, name):
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/experiment_api.py b/submarine-sdk/pysubmarine/submarine/client/api/experiment_api.py
index 01bac1cf..edb22c7a 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/experiment_api.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/experiment_api.py
@@ -24,20 +24,11 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
-from __future__ import absolute_import
-
-import re # noqa: F401
-
-# python 2 and python 3 compatibility library
-import six
-
from submarine.client.api_client import ApiClient
from submarine.client.exceptions import ApiTypeError, ApiValueError # noqa: F401
-class ExperimentApi(object):
+class ExperimentApi:
"""NOTE: This class is auto generated by OpenAPI Generator
Ref: https://openapi-generator.tech
@@ -100,15 +91,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["experiment_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method create_experiment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method create_experiment")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -208,15 +195,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method delete_experiment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method delete_experiment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -318,15 +301,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method get_experiment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method get_experiment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -428,24 +407,18 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method get_log" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method get_log")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
if self.api_client.client_side_validation and (
"id" not in local_var_params or local_var_params["id"] is None # noqa: E501
): # noqa: E501
- raise ApiValueError(
- "Missing the required parameter `id` when calling `get_log`"
- ) # noqa: E501
+ raise ApiValueError("Missing the required parameter `id` when calling `get_log`") # noqa: E501
collection_formats = {}
@@ -536,14 +509,12 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = []
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method get_tensorboard_info" % key
+ f"Got an unexpected keyword argument '{key}' to method get_tensorboard_info"
)
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -637,15 +608,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["status"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method list_experiments" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method list_experiments")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -740,15 +707,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["status"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method list_log" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method list_log")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -845,15 +808,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["id", "experiment_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method patch_experiment" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method patch_experiment")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -962,13 +921,11 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = []
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError("Got an unexpected keyword argument '%s' to method ping" % key)
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method ping")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -1063,15 +1020,12 @@ class ExperimentApi(object):
local_var_params = locals()
all_params = ["name", "experiment_template_submit"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method submit_experiment_template"
- % key
+ f"Got an unexpected keyword argument '{key}' to method submit_experiment_template"
)
local_var_params[key] = val
del local_var_params["kwargs"]
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/notebook_api.py b/submarine-sdk/pysubmarine/submarine/client/api/notebook_api.py
index 173bf849..c9a28995 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/notebook_api.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/notebook_api.py
@@ -24,20 +24,11 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
-from __future__ import absolute_import
-
-import re # noqa: F401
-
-# python 2 and python 3 compatibility library
-import six
-
from submarine.client.api_client import ApiClient
-from submarine.client.exceptions import ApiTypeError, ApiValueError # noqa: F401
+from submarine.client.exceptions import ApiTypeError, ApiValueError
-class NotebookApi(object):
+class NotebookApi:
"""NOTE: This class is auto generated by OpenAPI Generator
Ref: https://openapi-generator.tech
@@ -100,15 +91,11 @@ class NotebookApi(object):
local_var_params = locals()
all_params = ["notebook_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method create_notebook" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method create_notebook")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -208,15 +195,11 @@ class NotebookApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method delete_notebook" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method delete_notebook")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -318,15 +301,11 @@ class NotebookApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method get_notebook" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method get_notebook")
local_var_params[key] = val
del local_var_params["kwargs"]
# verify the required parameter 'id' is set
@@ -428,15 +407,11 @@ class NotebookApi(object):
local_var_params = locals()
all_params = ["id"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method list_notebooks" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method list_notebooks")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -531,13 +506,11 @@ class NotebookApi(object):
local_var_params = locals()
all_params = []
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError("Got an unexpected keyword argument '%s' to method ping1" % key)
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method ping1")
local_var_params[key] = val
del local_var_params["kwargs"]
diff --git a/submarine-sdk/pysubmarine/submarine/client/api/serve_api.py b/submarine-sdk/pysubmarine/submarine/client/api/serve_api.py
index 9666d563..af94725b 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api/serve_api.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api/serve_api.py
@@ -24,20 +24,11 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
-from __future__ import absolute_import
-
-import re # noqa: F401
-
-# python 2 and python 3 compatibility library
-import six
-
from submarine.client.api_client import ApiClient
-from submarine.client.exceptions import ApiTypeError, ApiValueError # noqa: F401
+from submarine.client.exceptions import ApiTypeError
-class ServeApi(object):
+class ServeApi:
"""NOTE: This class is auto generated by OpenAPI Generator
Ref: https://openapi-generator.tech
@@ -100,15 +91,11 @@ class ServeApi(object):
local_var_params = locals()
all_params = ["serve_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method create_serve" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method create_serve")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -208,15 +195,11 @@ class ServeApi(object):
local_var_params = locals()
all_params = ["serve_spec"]
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError(
- "Got an unexpected keyword argument '%s' to method delete_serve" % key
- )
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method delete_serve")
local_var_params[key] = val
del local_var_params["kwargs"]
@@ -311,13 +294,11 @@ class ServeApi(object):
local_var_params = locals()
all_params = []
- all_params.extend(
- ["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"]
- )
+ all_params.extend(["async_req", "_return_http_data_only", "_preload_content", "_request_timeout"])
- for key, val in six.iteritems(local_var_params["kwargs"]):
+ for key, val in local_var_params["kwargs"].items():
if key not in all_params:
- raise ApiTypeError("Got an unexpected keyword argument '%s' to method ping2" % key)
+ raise ApiTypeError(f"Got an unexpected keyword argument '{key}' to method ping2")
local_var_params[key] = val
del local_var_params["kwargs"]
diff --git a/submarine-sdk/pysubmarine/submarine/client/api_client.py b/submarine-sdk/pysubmarine/submarine/client/api_client.py
index e51ac65f..72d53156 100644
--- a/submarine-sdk/pysubmarine/submarine/client/api_client.py
+++ b/submarine-sdk/pysubmarine/submarine/client/api_client.py
@@ -24,7 +24,6 @@
Generated by: https://openapi-generator.tech
"""
-from __future__ import absolute_import
import atexit
import datetime
@@ -34,11 +33,9 @@ import os
import re
import tempfile
from multiprocessing.pool import ThreadPool
+from urllib.parse import quote
-# python 2 and python 3 compatibility library
-import six
from dateutil.parser import parse
-from six.moves.urllib.parse import quote
import submarine.client.models
from submarine.client import rest
@@ -46,7 +43,7 @@ from submarine.client.configuration import Configuration
from submarine.client.exceptions import ApiException, ApiValueError
-class ApiClient(object):
+class ApiClient:
"""Generic API client for OpenAPI client library builds.
OpenAPI generic API client. This client handles the client-
@@ -68,7 +65,7 @@ class ApiClient(object):
to the API. More threads means more concurrent API requests.
"""
- PRIMITIVE_TYPES = (float, bool, bytes, six.text_type) + six.integer_types
+ PRIMITIVE_TYPES = (bool, bytes, float, int, str)
NATIVE_TYPES_MAPPING = {
"int": int,
"long": int,
@@ -81,9 +78,7 @@ class ApiClient(object):
}
_pool = None
- def __init__(
- self, configuration=None, header_name=None, header_value=None, cookie=None, pool_threads=1
- ):
+ def __init__(self, configuration=None, header_name=None, header_value=None, cookie=None, pool_threads=1):
if configuration is None:
configuration = Configuration.get_default_copy()
self.configuration = configuration
@@ -152,7 +147,6 @@ class ApiClient(object):
_request_timeout=None,
_host=None,
):
-
config = self.configuration
# header parameters
@@ -213,7 +207,7 @@ class ApiClient(object):
_request_timeout=_request_timeout,
)
except ApiException as e:
- e.body = e.body.decode("utf-8") if six.PY3 else e.body
+ e.body = e.body.decode("utf-8")
raise e
content_type = response_data.getheader("content-type")
@@ -225,7 +219,7 @@ class ApiClient(object):
if not _preload_content:
return return_data
- if six.PY3 and response_type not in ["file", "bytes"]:
+ if response_type not in ["file", "bytes"]:
match = None
if content_type is not None:
match = re.search(r"charset=([a-zA-Z\-\d]+)[\s\;]?", content_type)
@@ -278,11 +272,11 @@ class ApiClient(object):
# model definition for request.
obj_dict = {
obj.attribute_map[attr]: getattr(obj, attr)
- for attr, _ in six.iteritems(obj.openapi_types)
+ for attr, _ in obj.openapi_types.items()
if getattr(obj, attr) is not None
}
- return {key: self.sanitize_for_serialization(val) for key, val in six.iteritems(obj_dict)}
+ return {key: self.sanitize_for_serialization(val) for key, val in obj_dict.items()}
def deserialize(self, response, response_type):
"""Deserializes response into an object.
@@ -324,7 +318,7 @@ class ApiClient(object):
if klass.startswith("dict("):
sub_kls = re.match(r"dict\(([^,]*), (.*)\)", klass).group(2)
- return {k: self.__deserialize(v, sub_kls) for k, v in six.iteritems(data)}
+ return {k: self.__deserialize(v, sub_kls) for k, v in data.items()}
# convert str to class
if klass in self.NATIVE_TYPES_MAPPING:
@@ -528,7 +522,7 @@ class ApiClient(object):
new_params = []
if collection_formats is None:
collection_formats = {}
- for k, v in six.iteritems(params) if isinstance(params, dict) else params: # noqa: E501
+ for k, v in params.items() if isinstance(params, dict) else params: # noqa: E501
if k in collection_formats:
collection_format = collection_formats[k]
if collection_format == "multi":
@@ -556,7 +550,7 @@ class ApiClient(object):
params = []
if files:
- for k, v in six.iteritems(files):
+ for k, v in files.items():
if not v:
continue
file_names = v if type(v) is list else [v]
@@ -657,7 +651,7 @@ class ApiClient(object):
try:
return klass(data)
except UnicodeEncodeError:
- return six.text_type(data)
+ return str(data)
except TypeError:
return data
@@ -679,9 +673,7 @@ class ApiClient(object):
except ImportError:
return string
except ValueError:
- raise rest.ApiException(
- status=0, reason="Failed to parse `{0}` as date object".format(string)
- )
+ raise rest.ApiException(status=0, reason=f"Failed to parse `{string}` as date object")
def __deserialize_datetime(self, string):
"""Deserializes string to datetime.
@@ -696,9 +688,7 @@ class ApiClient(object):
except ImportError:
return string
except ValueError:
- raise rest.ApiException(
- status=0, reason="Failed to parse `{0}` as datetime object".format(string)
- )
+ raise rest.ApiException(status=0, reason=f"Failed to parse `{string}` as datetime object")
def __deserialize_model(self, data, klass):
"""Deserializes list or dict to model.
@@ -716,7 +706,7 @@ class ApiClient(object):
kwargs = {}
if data is not None and klass.openapi_types is not None and isinstance(data, (list, dict)):
- for attr, attr_type in six.iteritems(klass.openapi_types):
+ for attr, attr_type in klass.openapi_types.items():
if klass.attribute_map[attr] in data:
value = data[klass.attribute_map[attr]]
kwargs[attr] = self.__deserialize(value, attr_type)
diff --git a/submarine-sdk/pysubmarine/submarine/client/configuration.py b/submarine-sdk/pysubmarine/submarine/client/configuration.py
index d5686f2b..0edf4fd4 100644
--- a/submarine-sdk/pysubmarine/submarine/client/configuration.py
+++ b/submarine-sdk/pysubmarine/submarine/client/configuration.py
@@ -26,19 +26,16 @@
"""
-from __future__ import absolute_import
-
import copy
import logging
import multiprocessing
import sys
+from http import client as httplib
-import six
import urllib3
-from six.moves import http_client as httplib
-class Configuration(object):
+class Configuration:
"""NOTE: This class is auto generated by OpenAPI Generator
Ref: https://openapi-generator.tech
@@ -239,7 +236,7 @@ class Configuration(object):
# then add file handler and remove stream handler.
self.logger_file_handler = logging.FileHandler(self.__logger_file)
self.logger_file_handler.setFormatter(self.logger_formatter)
- for _, logger in six.iteritems(self.logger):
+ for _, logger in self.logger.items():
logger.addHandler(self.logger_file_handler)
@property
@@ -261,14 +258,14 @@ class Configuration(object):
self.__debug = value
if self.__debug:
# if debug status is True, turn on debug logging
- for _, logger in six.iteritems(self.logger):
+ for _, logger in self.logger.items():
logger.setLevel(logging.DEBUG)
# turn on httplib debug
httplib.HTTPConnection.debuglevel = 1
else:
# if debug status is False, turn off debug logging,
# setting log level to default `logging.WARNING`
- for _, logger in six.iteritems(self.logger):
+ for _, logger in self.logger.items():
logger.setLevel(logging.WARNING)
# turn off httplib debug
httplib.HTTPConnection.debuglevel = 0
@@ -308,7 +305,7 @@ class Configuration(object):
if key:
prefix = self.api_key_prefix.get(identifier)
if prefix:
- return "%s %s" % (prefix, key)
+ return f"{prefix} {key}"
else:
return key
@@ -340,10 +337,10 @@ class Configuration(object):
"""
return (
"Python SDK Debug Report:\n"
- "OS: {env}\n"
- "Python Version: {pyversion}\n"
+ f"OS: {sys.platform}\n"
+ f"Python Version: {sys.version}\n"
"Version of the API: 0.8.0-SNAPSHOT\n"
- "SDK Package Version: 0.8.0-SNAPSHOT".format(env=sys.platform, pyversion=sys.version)
+ "SDK Package Version: 0.8.0-SNAPSHOT"
)
def get_host_settings(self):
@@ -371,9 +368,7 @@ class Configuration(object):
server = servers[index]
except IndexError:
raise ValueError(
- "Invalid index {0} when selecting the host settings. Must be less than {1}".format(
- index, len(servers)
- )
+ f"Invalid index {index} when selecting the host settings. Must be less than {len(servers)}"
)
url = server["url"]
@@ -384,9 +379,8 @@ class Configuration(object):
if "enum_values" in variable and used_value not in variable["enum_values"]:
raise ValueError(
- "The variable `{0}` in the host URL has invalid value {1}. Must be {2}.".format(
- variable_name, variables[variable_name], variable["enum_values"]
- )
+ f"The variable `{variable_name}` in the host URL has invalid value"
+ f" {variables[variable_name]}. Must be {variable['enum_values']}."
)
url = url.replace("{" + variable_name + "}", used_value)
diff --git a/submarine-sdk/pysubmarine/submarine/client/exceptions.py b/submarine-sdk/pysubmarine/submarine/client/exceptions.py
index b714f7a2..4f04307d 100644
--- a/submarine-sdk/pysubmarine/submarine/client/exceptions.py
+++ b/submarine-sdk/pysubmarine/submarine/client/exceptions.py
@@ -26,9 +26,6 @@
"""
-import six
-
-
class OpenApiException(Exception):
"""The base exception class for all OpenAPIExceptions"""
@@ -57,8 +54,8 @@ class ApiTypeError(OpenApiException, TypeError):
self.key_type = key_type
full_msg = msg
if path_to_item:
- full_msg = "{0} at {1}".format(msg, render_path(path_to_item))
- super(ApiTypeError, self).__init__(full_msg)
+ full_msg = f"{msg} at {render_path(path_to_item)}"
+ super().__init__(full_msg)
class ApiValueError(OpenApiException, ValueError):
@@ -75,8 +72,8 @@ class ApiValueError(OpenApiException, ValueError):
self.path_to_item = path_to_item
full_msg = msg
if path_to_item:
- full_msg = "{0} at {1}".format(msg, render_path(path_to_item))
- super(ApiValueError, self).__init__(full_msg)
+ full_msg = f"{msg} at {render_path(path_to_item)}"
+ super().__init__(full_msg)
class ApiKeyError(OpenApiException, KeyError):
@@ -92,8 +89,8 @@ class ApiKeyError(OpenApiException, KeyError):
self.path_to_item = path_to_item
full_msg = msg
if path_to_item:
- full_msg = "{0} at {1}".format(msg, render_path(path_to_item))
- super(ApiKeyError, self).__init__(full_msg)
+ full_msg = f"{msg} at {render_path(path_to_item)}"
+ super().__init__(full_msg)
class ApiException(OpenApiException):
@@ -111,12 +108,12 @@ class ApiException(OpenApiException):
def __str__(self):
"""Custom error messages for exception"""
- error_message = "({0})\nReason: {1}\n".format(self.status, self.reason)
+ error_message = f"({self.status})\nReason: {self.reason}\n"
if self.headers:
- error_message += "HTTP response headers: {0}\n".format(self.headers)
+ error_message += f"HTTP response headers: {self.headers}\n"
if self.body:
- error_message += "HTTP response body: {0}\n".format(self.body)
+ error_message += f"HTTP response body: {self.body}\n"
return error_message
@@ -125,8 +122,8 @@ def render_path(path_to_item):
"""Returns a string representation of a path"""
result = ""
for pth in path_to_item:
- if isinstance(pth, six.integer_types):
- result += "[{0}]".format(pth)
+ if isinstance(pth, int):
+ result += f"[{pth}]"
else:
- result += "['{0}']".format(pth)
+ result += f"['{pth}']"
return result
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/__init__.py b/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
index 1996255c..07352e4f 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
@@ -27,8 +27,6 @@
"""
-from __future__ import absolute_import
-
# import models into model package
from submarine.client.models.code_spec import CodeSpec
from submarine.client.models.environment_spec import EnvironmentSpec
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
index 32551c3d..1a9e44d9 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class CodeSpec(object):
+class CodeSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -113,20 +110,16 @@ class CodeSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/environment_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/environment_spec.py
index 654f9988..7fd254cd 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/environment_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/environment_spec.py
@@ -24,17 +24,12 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class EnvironmentSpec(object):
+class EnvironmentSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -205,20 +200,16 @@ class EnvironmentSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/experiment_meta.py b/submarine-sdk/pysubmarine/submarine/client/models/experiment_meta.py
index dabd7127..2139da77 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/experiment_meta.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/experiment_meta.py
@@ -24,17 +24,12 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ExperimentMeta(object):
+class ExperimentMeta:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -259,20 +254,16 @@ class ExperimentMeta(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/experiment_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/experiment_spec.py
index 0175745e..f9c62406 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/experiment_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/experiment_spec.py
@@ -24,17 +24,12 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ExperimentSpec(object):
+class ExperimentSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -168,20 +163,16 @@ class ExperimentSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
index 722b821d..bd861dae 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ExperimentTaskSpec(object):
+class ExperimentTaskSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -313,20 +310,16 @@ class ExperimentTaskSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/experiment_template_submit.py b/submarine-sdk/pysubmarine/submarine/client/models/experiment_template_submit.py
index 1f1f8ca9..49c7d465 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/experiment_template_submit.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/experiment_template_submit.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ExperimentTemplateSubmit(object):
+class ExperimentTemplateSubmit:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -113,20 +110,16 @@ class ExperimentTemplateSubmit(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/json_response.py b/submarine-sdk/pysubmarine/submarine/client/models/json_response.py
index 06f48327..6040d414 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/json_response.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/json_response.py
@@ -24,17 +24,12 @@
Contact: dev@submarine.apache.org
Generated by: https://openapi-generator.tech
"""
-
-
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class JsonResponse(object):
+class JsonResponse:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -173,20 +168,16 @@ class JsonResponse(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/kernel_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/kernel_spec.py
index 9e17eda2..fedfa7b0 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/kernel_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/kernel_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class KernelSpec(object):
+class KernelSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -178,20 +175,16 @@ class KernelSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/notebook_meta.py b/submarine-sdk/pysubmarine/submarine/client/models/notebook_meta.py
index 201b8129..e801281a 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/notebook_meta.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/notebook_meta.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class NotebookMeta(object):
+class NotebookMeta:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -173,20 +170,16 @@ class NotebookMeta(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
index bac767d2..31c66069 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class NotebookPodSpec(object):
+class NotebookPodSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -205,20 +202,16 @@ class NotebookPodSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/notebook_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/notebook_spec.py
index 9f184827..219d80cd 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/notebook_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/notebook_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class NotebookSpec(object):
+class NotebookSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -56,9 +53,7 @@ class NotebookSpec(object):
attribute_map = {"meta": "meta", "environment": "environment", "spec": "spec"}
- def __init__(
- self, meta=None, environment=None, spec=None, local_vars_configuration=None
- ): # noqa: E501
+ def __init__(self, meta=None, environment=None, spec=None, local_vars_configuration=None): # noqa: E501
"""NotebookSpec - a model defined in OpenAPI""" # noqa: E501
if local_vars_configuration is None:
local_vars_configuration = Configuration()
@@ -143,20 +138,16 @@ class NotebookSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/serve_request.py b/submarine-sdk/pysubmarine/submarine/client/models/serve_request.py
index 0fcde78d..6b36a23e 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/serve_request.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/serve_request.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ServeRequest(object):
+class ServeRequest:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -143,20 +140,16 @@ class ServeRequest(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/serve_spec.py b/submarine-sdk/pysubmarine/submarine/client/models/serve_spec.py
index e9b622c7..7151343e 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/serve_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/serve_spec.py
@@ -27,14 +27,11 @@
import pprint
-import re # noqa: F401
-
-import six
from submarine.client.configuration import Configuration
-class ServeSpec(object):
+class ServeSpec:
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -178,20 +175,16 @@ class ServeSpec(object):
"""Returns the model properties as a dict"""
result = {}
- for attr, _ in six.iteritems(self.openapi_types):
+ for attr, _ in self.openapi_types.items():
value = getattr(self, attr)
if isinstance(value, list):
- result[attr] = list(
- map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
- )
+ result[attr] = list(map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value))
elif hasattr(value, "to_dict"):
result[attr] = value.to_dict()
elif isinstance(value, dict):
result[attr] = dict(
map(
- lambda item: (item[0], item[1].to_dict())
- if hasattr(item[1], "to_dict")
- else item,
+ lambda item: (item[0], item[1].to_dict()) if hasattr(item[1], "to_dict") else item,
value.items(),
)
)
diff --git a/submarine-sdk/pysubmarine/submarine/client/rest.py b/submarine-sdk/pysubmarine/submarine/client/rest.py
index 4fe7fc05..4e2039cd 100644
--- a/submarine-sdk/pysubmarine/submarine/client/rest.py
+++ b/submarine-sdk/pysubmarine/submarine/client/rest.py
@@ -26,20 +26,15 @@
"""
-from __future__ import absolute_import
-
import io
import json
import logging
import re
import ssl
+from urllib.parse import urlencode
import certifi
-
-# python 2 and python 3 compatibility library
-import six
import urllib3
-from six.moves.urllib.parse import urlencode
from submarine.client.exceptions import ApiException, ApiValueError
@@ -62,7 +57,7 @@ class RESTResponse(io.IOBase):
return self.urllib3_response.getheader(name, default)
-class RESTClientObject(object):
+class RESTClientObject:
def __init__(self, configuration, pools_size=4, maxsize=None):
# urllib3.PoolManager will pass all kw parameters to connectionpool
# https://github.com/shazow/urllib3/blob/f9409436f83aeb79fbaf090181cd81b784f1b8ce/urllib3/poolmanager.py#L75 # noqa: E501
@@ -107,7 +102,7 @@ class RESTClientObject(object):
key_file=configuration.key_file,
proxy_url=configuration.proxy,
proxy_headers=configuration.proxy_headers,
- **addition_pool_args
+ **addition_pool_args,
)
else:
self.pool_manager = urllib3.PoolManager(
@@ -117,7 +112,7 @@ class RESTClientObject(object):
ca_certs=ca_certs,
cert_file=configuration.cert_file,
key_file=configuration.key_file,
- **addition_pool_args
+ **addition_pool_args,
)
def request(
@@ -160,7 +155,7 @@ class RESTClientObject(object):
timeout = None
if _request_timeout:
- if isinstance(_request_timeout, (int,) if six.PY3 else (int, long)): # noqa: E501,F821
+ if isinstance(_request_timeout, (int,)):
timeout = urllib3.Timeout(total=_request_timeout)
elif isinstance(_request_timeout, tuple) and len(_request_timeout) == 2:
timeout = urllib3.Timeout(connect=_request_timeout[0], read=_request_timeout[1])
@@ -239,7 +234,7 @@ class RESTClientObject(object):
headers=headers,
)
except urllib3.exceptions.SSLError as e:
- msg = "{0}\n{1}".format(type(e).__name__, str(e))
+ msg = f"{type(e).__name__}\n{str(e)}"
raise ApiException(status=0, reason=msg)
if _preload_content:
@@ -253,9 +248,7 @@ class RESTClientObject(object):
return r
- def GET(
- self, url, headers=None, query_params=None, _preload_content=True, _request_timeout=None
- ):
+ def GET(self, url, headers=None, query_params=None, _preload_content=True, _request_timeout=None):
return self.request(
"GET",
url,
@@ -265,9 +258,7 @@ class RESTClientObject(object):
query_params=query_params,
)
- def HEAD(
- self, url, headers=None, query_params=None, _preload_content=True, _request_timeout=None
- ):
+ def HEAD(self, url, headers=None, query_params=None, _preload_content=True, _request_timeout=None):
return self.request(
"HEAD",
url,
diff --git a/submarine-sdk/pysubmarine/submarine/entities/_submarine_object.py b/submarine-sdk/pysubmarine/submarine/entities/_submarine_object.py
index db2ad094..015207f0 100644
--- a/submarine-sdk/pysubmarine/submarine/entities/_submarine_object.py
+++ b/submarine-sdk/pysubmarine/submarine/entities/_submarine_object.py
@@ -24,7 +24,7 @@ class _SubmarineObject:
@classmethod
def _properties(cls):
- return sorted([p for p in cls.__dict__ if isinstance(getattr(cls, p), property)])
+ return sorted(p for p in cls.__dict__ if isinstance(getattr(cls, p), property))
@classmethod
def from_dictionary(cls, the_dict):
@@ -45,13 +45,13 @@ def get_classname(obj):
class _SubmarineObjectPrinter:
def __init__(self):
- super(_SubmarineObjectPrinter, self).__init__()
+ super().__init__()
self.printer = pprint.PrettyPrinter()
def to_string(self, obj) -> str:
if isinstance(obj, _SubmarineObject):
- return "<%s: %s>" % (get_classname(obj), self._entity_to_string(obj))
+ return f"<{get_classname(obj)}: {self._entity_to_string(obj)}>"
return self.printer.pformat(obj)
def _entity_to_string(self, entity) -> str:
- return ", ".join(["%s=%s" % (key, self.to_string(value)) for key, value in entity])
+ return ", ".join([f"{key}={self.to_string(value)}" for key, value in entity])
diff --git a/submarine-sdk/pysubmarine/submarine/exceptions.py b/submarine-sdk/pysubmarine/submarine/exceptions.py
index d6c5c4d3..06ecd7df 100644
--- a/submarine-sdk/pysubmarine/submarine/exceptions.py
+++ b/submarine-sdk/pysubmarine/submarine/exceptions.py
@@ -24,7 +24,7 @@ class SubmarineException(Exception):
:param message: The message describing the error that occurred.
"""
self.message = message
- super(SubmarineException, self).__init__(message)
+ super().__init__(message)
class RestException(SubmarineException):
@@ -32,9 +32,7 @@ class RestException(SubmarineException):
def __init__(self, json):
error_code = json.get("error_code")
- message = "%s: %s" % (
- error_code,
- json["message"] if "message" in json else "Response: " + str(json),
- )
- super(RestException, self).__init__(message)
+ error_msg = json["message"] if "message" in json else "Response: " + str(json)
+ message = f"{error_code}: {error_msg}"
+ super().__init__(message)
self.json = json
diff --git a/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py b/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
index 76efa3b1..957cad82 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
@@ -85,9 +85,7 @@ class LIBSVMDataset(Dataset):
)
@classmethod
- def _locate_sample_offsets_job(
- cls, data_uri: str, task: Tuple[int, Tuple[int, int]]
- ) -> List[int]:
+ def _locate_sample_offsets_job(cls, data_uri: str, task: Tuple[int, Tuple[int, int]]) -> List[int]:
_, (start, end) = task
offsets = [start]
with open_buffered_file_reader(data_uri) as infile:
diff --git a/submarine-sdk/pysubmarine/submarine/ml/pytorch/layers/core.py b/submarine-sdk/pysubmarine/submarine/ml/pytorch/layers/core.py
index fd7d9e94..3e1162a4 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/layers/core.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/layers/core.py
@@ -33,9 +33,7 @@ class FeatureLinear(nn.Module):
:param feature_idx: torch.LongTensor (batch_size, num_fields)
:param feature_value: torch.LongTensor (batch_size, num_fields)
"""
- return (
- torch.sum(self.weight(feature_idx) * feature_value.unsqueeze(dim=-1), dim=1) + self.bias
- )
+ return torch.sum(self.weight(feature_idx) * feature_value.unsqueeze(dim=-1), dim=1) + self.bias
class FeatureEmbedding(nn.Module):
@@ -59,9 +57,7 @@ class PairwiseInteraction(nn.Module):
square_of_sum = torch.square(torch.sum(x, dim=1)) # (batch_size, embedding_dim)
# (batch_size, embedding_dim)
sum_of_square = torch.sum(torch.square(x), dim=1)
- return 0.5 * torch.sum(
- square_of_sum - sum_of_square, dim=1, keepdim=True
- ) # (batch_size, 1)
+ return 0.5 * torch.sum(square_of_sum - sum_of_square, dim=1, keepdim=True) # (batch_size, 1)
class DNN(nn.Module):
@@ -78,7 +74,7 @@ class DNN(nn.Module):
)
for (i, o), p in zip(layers, dropout_rates)
),
- nn.Linear(*out_layer)
+ nn.Linear(*out_layer),
)
def forward(self, x: torch.FloatTensor):
diff --git a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/base_pytorch_model.py b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/base_pytorch_model.py
index 846241d4..771336c1 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/base_pytorch_model.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/base_pytorch_model.py
@@ -95,9 +95,7 @@ class BasePyTorchModel(AbstractModel, ABC):
outputs.append(output)
labels.append(label)
- return self.metric(
- torch.cat(labels, dim=0).cpu().numpy(), torch.cat(outputs, dim=0).cpu().numpy()
- )
+ return self.metric(torch.cat(labels, dim=0).cpu().numpy(), torch.cat(outputs, dim=0).cpu().numpy())
def predict(self):
outputs = []
@@ -142,9 +140,7 @@ class BasePyTorchModel(AbstractModel, ABC):
{"model": self.model.module.state_dict(), "optimizer": self.optimizer.state_dict()},
buffer,
)
- write_file(
- buffer, uri=os.path.join(self.params["output"]["save_model_dir"], "ckpt.pkl")
- )
+ write_file(buffer, uri=os.path.join(self.params["output"]["save_model_dir"], "ckpt.pkl"))
def model_fn(self, params):
seed = params["training"]["seed"]
diff --git a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/afm.py b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/afm.py
index 10625481..0e311d80 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/afm.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/afm.py
@@ -35,13 +35,11 @@ class _AFM(nn.Module):
attention_dim: int,
out_features: int,
dropout_rate: float,
- **kwargs
+ **kwargs,
):
super().__init__()
self.feature_linear = FeatureLinear(num_features=num_features, out_features=out_features)
- self.feature_embedding = FeatureEmbedding(
- num_features=num_features, embedding_dim=embedding_dim
- )
+ self.feature_embedding = FeatureEmbedding(num_features=num_features, embedding_dim=embedding_dim)
self.attentional_interaction = AttentionalInteratction(
embedding_dim=embedding_dim,
attention_dim=attention_dim,
@@ -60,9 +58,7 @@ class _AFM(nn.Module):
class AttentionalInteratction(nn.Module):
- def __init__(
- self, embedding_dim: int, attention_dim: int, out_features: int, dropout_rate: float
- ):
+ def __init__(self, embedding_dim: int, attention_dim: int, out_features: int, dropout_rate: float):
super().__init__()
self.attention_score = nn.Sequential(
nn.Linear(in_features=embedding_dim, out_features=attention_dim),
diff --git a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/deepfm.py b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/deepfm.py
index 6bc97604..92e642ec 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/deepfm.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/model/ctr/deepfm.py
@@ -41,13 +41,11 @@ class _DeepFM(nn.Module):
out_features,
hidden_units,
dropout_rates,
- **kwargs
+ **kwargs,
):
super().__init__()
self.feature_linear = FeatureLinear(num_features=num_features, out_features=out_features)
- self.feature_embedding = FeatureEmbedding(
- num_features=num_features, embedding_dim=embedding_dim
- )
+ self.feature_embedding = FeatureEmbedding(num_features=num_features, embedding_dim=embedding_dim)
self.pairwise_interaction = PairwiseInteraction()
self.dnn = DNN(
in_features=num_fields * embedding_dim,
@@ -61,9 +59,7 @@ class _DeepFM(nn.Module):
:param feature_idx: torch.LongTensor (batch_size, num_fields)
:param feature_value: torch.LongTensor (batch_size, num_fields)
"""
- emb = self.feature_embedding(
- feature_idx, feature_value
- ) # (batch_size, num_fields, embedding_dim)
+ emb = self.feature_embedding(feature_idx, feature_value) # (batch_size, num_fields, embedding_dim)
linear_logit = self.feature_linear(feature_idx, feature_value)
fm_logit = self.pairwise_interaction(emb)
deep_logit = self.dnn(torch.flatten(emb, start_dim=1))
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/input/input.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/input/input.py
index 0cc72596..d61f59fa 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/input/input.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/input/input.py
@@ -28,7 +28,7 @@ def libsvm_input_fn(
num_epochs: int = 3, # pylint: disable=W0613
perform_shuffle: bool = False,
delimiter: str = " ",
- **kwargs
+ **kwargs,
):
def _input_fn():
def decode_libsvm(line):
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/layers/core.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/layers/core.py
index 47f3698a..398906d2 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/layers/core.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/layers/core.py
@@ -49,7 +49,7 @@ def dnn_layer(
dropout,
batch_norm_decay: float = 0.9,
l2_reg: float = 0,
- **kwargs
+ **kwargs,
):
"""
The Multi Layer Perceptron
@@ -116,9 +116,7 @@ def linear_layer(features, feature_size, field_size, l2_reg: float = 0, **kwargs
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
with tf.variable_scope("LinearLayer_Layer"):
- linear_bias = tf.get_variable(
- name="linear_bias", shape=[1], initializer=tf.constant_initializer(0.0)
- )
+ linear_bias = tf.get_variable(name="linear_bias", shape=[1], initializer=tf.constant_initializer(0.0))
linear_weight = tf.get_variable(
name="linear_weight",
shape=[feature_size],
@@ -131,9 +129,7 @@ def linear_layer(features, feature_size, field_size, l2_reg: float = 0, **kwargs
return linear_out
-def embedding_layer(
- features, feature_size, field_size, embedding_size, l2_reg: float = 0, **kwargs
-):
+def embedding_layer(features, feature_size, field_size, embedding_size, l2_reg: float = 0, **kwargs):
"""
Turns positive integers (indexes) into dense vectors of fixed size.
eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
@@ -202,24 +198,21 @@ class KMaxPooling(Layer):
"""
def __init__(self, k: int = 1, axis: int = -1, **kwargs):
-
self.dims = 1
self.k = k
self.axis = axis
- super(KMaxPooling, self).__init__(**kwargs)
+ super().__init__(**kwargs)
def build(self, input_shape):
-
if self.axis < 1 or self.axis > len(input_shape):
raise ValueError("axis must be 1~%d,now is %d" % (len(input_shape), self.axis))
if self.k < 1 or self.k > input_shape[self.axis]:
raise ValueError("k must be in 1 ~ %d,now k is %d" % (input_shape[self.axis], self.k))
self.dims = len(input_shape)
- super(KMaxPooling, self).build(input_shape)
+ super().build(input_shape)
def call(self, inputs):
-
perm = list(range(self.dims))
perm[-1], perm[self.axis] = perm[self.axis], perm[-1]
shifted_input = tf.transpose(inputs, perm)
@@ -238,5 +231,5 @@ class KMaxPooling(Layer):
self,
):
config = {"k": self.k, "axis": self.axis}
- base_config = super(KMaxPooling, self).get_config()
+ base_config = super().get_config()
return dict(list(base_config.items()) + list(config.items()))
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/model/deepfm.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/model/deepfm.py
index 64fb001b..34a87285 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/model/deepfm.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/model/deepfm.py
@@ -28,7 +28,12 @@ import logging
import tensorflow as tf
-from submarine.ml.tensorflow.layers.core import dnn_layer, embedding_layer, fm_layer, linear_layer
+from submarine.ml.tensorflow.layers.core import (
+ dnn_layer,
+ embedding_layer,
+ fm_layer,
+ linear_layer,
+)
from submarine.ml.tensorflow.model.base_tf_model import BaseTFModel
from submarine.utils.tf_utils import get_estimator_spec
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/optimizer.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/optimizer.py
index 3ab9bbb0..0bb7b14a 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow/optimizer.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow/optimizer.py
@@ -20,7 +20,7 @@ import tensorflow as tf
logger = logging.getLogger(__name__)
-class OptimizerKey(object):
+class OptimizerKey:
"""Optimizer key strings."""
ADAM = "adam"
@@ -33,9 +33,7 @@ def get_optimizer(optimizer_key: str, learning_rate: float):
optimizer_key = optimizer_key.lower()
if optimizer_key == OptimizerKey.ADAM:
- op = tf.train.AdamOptimizer(
- learning_rate=learning_rate, beta1=0.9, beta2=0.999, epsilon=1e-8
- )
+ op = tf.train.AdamOptimizer(learning_rate=learning_rate, beta1=0.9, beta2=0.999, epsilon=1e-8)
elif optimizer_key == OptimizerKey.ADAGRAD:
op = tf.train.AdagradOptimizer(learning_rate=learning_rate, initial_accumulator_value=1e-8)
elif optimizer_key == OptimizerKey.MOMENTUM:
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/input/input.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/input/input.py
index 0cf98ae4..dce0b291 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/input/input.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/input/input.py
@@ -28,7 +28,7 @@ def libsvm_input_fn(
num_epochs=3, # pylint: disable=W0613
perform_shuffle=False,
delimiter=" ",
- **kwargs
+ **kwargs,
):
def _input_fn():
def decode_libsvm(line):
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/layers/core.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/layers/core.py
index a75ff710..fef2135e 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/layers/core.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/layers/core.py
@@ -41,9 +41,7 @@ def batch_norm_layer(x, train_phase, scope_bn, batch_norm_decay):
reuse=True,
scope=scope_bn,
)
- return tf.cond(
- pred=tf.cast(train_phase, tf.bool), true_fn=lambda: bn_train, false_fn=lambda: bn_infer
- )
+ return tf.cond(pred=tf.cast(train_phase, tf.bool), true_fn=lambda: bn_train, false_fn=lambda: bn_infer)
def dnn_layer(
@@ -54,7 +52,7 @@ def dnn_layer(
dropout,
batch_norm_decay=0.9,
l2_reg=0,
- **kwargs
+ **kwargs,
):
"""
The Multi Layer Perceptron
@@ -72,7 +70,6 @@ def dnn_layer(
"""
with tf.compat.v1.variable_scope("DNN_Layer"):
if batch_norm:
-
if estimator_mode == tf.estimator.ModeKeys.TRAIN:
train_phase = True
else:
@@ -133,9 +130,7 @@ def linear_layer(features, feature_size, field_size, l2_reg=0, **kwargs):
)
feat_weights = tf.nn.embedding_lookup(params=linear_weight, ids=feat_ids)
- linear_out = (
- tf.reduce_sum(input_tensor=tf.multiply(feat_weights, feat_vals), axis=1) + linear_bias
- )
+ linear_out = tf.reduce_sum(input_tensor=tf.multiply(feat_weights, feat_vals), axis=1) + linear_bias
return linear_out
@@ -208,24 +203,21 @@ class KMaxPooling(Layer):
"""
def __init__(self, k=1, axis=-1, **kwargs):
-
self.dims = 1
self.k = k
self.axis = axis
- super(KMaxPooling, self).__init__(**kwargs)
+ super().__init__(**kwargs)
def build(self, input_shape):
-
if self.axis < 1 or self.axis > len(input_shape):
raise ValueError("axis must be 1~%d,now is %d" % (len(input_shape), self.axis))
if self.k < 1 or self.k > input_shape[self.axis]:
raise ValueError("k must be in 1 ~ %d,now k is %d" % (input_shape[self.axis], self.k))
self.dims = len(input_shape)
- super(KMaxPooling, self).build(input_shape)
+ super().build(input_shape)
def call(self, inputs):
-
perm = list(range(self.dims))
perm[-1], perm[self.axis] = perm[self.axis], perm[-1]
shifted_input = tf.transpose(a=inputs, perm=perm)
@@ -244,5 +236,5 @@ class KMaxPooling(Layer):
self,
):
config = {"k": self.k, "axis": self.axis}
- base_config = super(KMaxPooling, self).get_config()
+ base_config = super().get_config()
return dict(list(base_config.items()) + list(config.items()))
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/model/fm.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/model/fm.py
index e94a15e8..a992161d 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/model/fm.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/model/fm.py
@@ -24,7 +24,11 @@ import logging
import tensorflow as tf
-from submarine.ml.tensorflow_v2.layers.core import embedding_layer, fm_layer, linear_layer
+from submarine.ml.tensorflow_v2.layers.core import (
+ embedding_layer,
+ fm_layer,
+ linear_layer,
+)
from submarine.ml.tensorflow_v2.model.base_tf_model import BaseTFModel
from submarine.utils.tf_utils_v2 import get_estimator_spec
diff --git a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/optimizer.py b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/optimizer.py
index bdba587e..edcf6bcf 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/optimizer.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/tensorflow_v2/optimizer.py
@@ -20,7 +20,7 @@ import tensorflow as tf
logger = logging.getLogger(__name__)
-class OptimizerKey(object):
+class OptimizerKey:
"""Optimizer key strings."""
ADAM = "adam"
@@ -37,9 +37,7 @@ def get_optimizer(optimizer_key, learning_rate):
learning_rate=learning_rate, beta1=0.9, beta2=0.999, epsilon=1e-8
)
elif optimizer_key == OptimizerKey.ADAGRAD:
- op = tf.compat.v1.train.AdagradOptimizer(
- learning_rate=learning_rate, initial_accumulator_value=1e-8
- )
+ op = tf.compat.v1.train.AdagradOptimizer(learning_rate=learning_rate, initial_accumulator_value=1e-8)
elif optimizer_key == OptimizerKey.MOMENTUM:
op = tf.compat.v1.train.MomentumOptimizer(learning_rate=learning_rate, momentum=0.95)
elif optimizer_key == OptimizerKey.FTRL:
diff --git a/submarine-sdk/pysubmarine/submarine/store/database/models.py b/submarine-sdk/pysubmarine/submarine/store/database/models.py
index 68f592d6..545b5a82 100644
--- a/submarine-sdk/pysubmarine/submarine/store/database/models.py
+++ b/submarine-sdk/pysubmarine/submarine/store/database/models.py
@@ -124,9 +124,7 @@ class SqlRegisteredModel(Base):
class SqlRegisteredModelTag(Base):
__tablename__ = "registered_model_tag"
- name = Column(
- String(256), ForeignKey("registered_model.name", onupdate="cascade", ondelete="cascade")
- )
+ name = Column(String(256), ForeignKey("registered_model.name", onupdate="cascade", ondelete="cascade"))
"""
Name of registered model: Part of *Primary Key* for ``registered_model_tag`` table.
Refer to name of ``registered_model`` table.
@@ -235,9 +233,7 @@ class SqlModelVersion(Base):
"""
# linked entities
- registered_model: SqlRegisteredModel = relationship(
- "SqlRegisteredModel", back_populates="model_versions"
- )
+ registered_model: SqlRegisteredModel = relationship("SqlRegisteredModel", back_populates="model_versions")
__table_args__ = (
PrimaryKeyConstraint("name", "version", name="model_version_pk"),
@@ -368,13 +364,9 @@ class SqlExperiment(Base):
__table_args__ = (PrimaryKeyConstraint("id"),)
def __repr__(self):
- return "<SqlMetric({}, {}, {}, {}, {}, {})>".format(
- self.id,
- self.experiment_spec,
- self.create_by,
- self.create_time,
- self.update_by,
- self.update_time,
+ return (
+ f"<SqlMetric({self.id}, {self.experiment_spec}, {self.create_by}, {self.create_time},"
+ f" {self.update_by}, {self.update_time})>"
)
def to_submarine_entity(self):
@@ -436,14 +428,10 @@ class SqlMetric(Base):
True if the value is in fact NaN.
"""
- __table_args__ = (
- PrimaryKeyConstraint("id", "key", "timestamp", "worker_index", name="metric_pk"),
- )
+ __table_args__ = (PrimaryKeyConstraint("id", "key", "timestamp", "worker_index", name="metric_pk"),)
def __repr__(self):
- return "<SqlMetric({}, {}, {}, {}, {})>".format(
- self.key, self.value, self.worker_index, self.timestamp, self.step
- )
+ return f"<SqlMetric({self.key}, {self.value}, {self.worker_index}, {self.timestamp}, {self.step})>"
def to_submarine_entity(self):
"""
@@ -493,7 +481,7 @@ class SqlParam(Base):
__table_args__ = (PrimaryKeyConstraint("id", "key", "worker_index", name="param_pk"),)
def __repr__(self):
- return "<SqlParam({}, {}, {})>".format(self.key, self.value, self.worker_index)
+ return f"<SqlParam({self.key}, {self.value}, {self.worker_index})>"
def to_submarine_entity(self):
"""
diff --git a/submarine-sdk/pysubmarine/submarine/store/model_registry/abstract_store.py b/submarine-sdk/pysubmarine/submarine/store/model_registry/abstract_store.py
index 2ed38527..24c60868 100644
--- a/submarine-sdk/pysubmarine/submarine/store/model_registry/abstract_store.py
+++ b/submarine-sdk/pysubmarine/submarine/store/model_registry/abstract_store.py
@@ -149,9 +149,7 @@ class AbstractStore:
pass
@abstractmethod
- def update_model_version_description(
- self, name: str, version: int, description: str
- ) -> ModelVersion:
+ def update_model_version_description(self, name: str, version: int, description: str) -> ModelVersion:
"""
Update description associated with the version of model in backend.
:param name: Registered model name.
diff --git a/submarine-sdk/pysubmarine/submarine/store/model_registry/sqlalchemy_store.py b/submarine-sdk/pysubmarine/submarine/store/model_registry/sqlalchemy_store.py
index 5444d990..d2da6c10 100644
--- a/submarine-sdk/pysubmarine/submarine/store/model_registry/sqlalchemy_store.py
+++ b/submarine-sdk/pysubmarine/submarine/store/model_registry/sqlalchemy_store.py
@@ -58,7 +58,7 @@ class SqlAlchemyStore(AbstractStore):
<https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls>`_
for format specifications. Submarine supports the dialects ``mysql``.
"""
- super(SqlAlchemyStore, self).__init__()
+ super().__init__()
self.db_uri = db_uri
self.db_type = extract_db_type_from_uri(db_uri)
@@ -168,9 +168,7 @@ class SqlAlchemyStore(AbstractStore):
session.flush()
return registered_model.to_submarine_entity()
except sqlalchemy.exc.IntegrityError as e:
- raise SubmarineException(
- f"Registered model (name={name}) already exists.\nError: {str(e)}"
- )
+ raise SubmarineException(f"Registered model (name={name}) already exists.\nError: {str(e)}")
@classmethod
def _get_sql_registered_model(
@@ -235,15 +233,11 @@ class SqlAlchemyStore(AbstractStore):
for sql_model_version in sql_registered_model.model_versions:
sql_model_version.name = new_name
sql_model_version.last_updated_time = update_time
- self._save_to_db(
- session, [sql_registered_model] + sql_registered_model.model_versions
- )
+ self._save_to_db(session, [sql_registered_model] + sql_registered_model.model_versions)
session.flush()
return sql_registered_model.to_submarine_entity()
except sqlalchemy.exc.IntegrityError as e:
- raise SubmarineException(
- f"Registered Model (name={name}) already exists. Error: {str(e)}"
- )
+ raise SubmarineException(f"Registered Model (name={name}) already exists. Error: {str(e)}")
def delete_registered_model(self, name: str) -> None:
"""
@@ -268,16 +262,14 @@ class SqlAlchemyStore(AbstractStore):
conditions = []
if filter_tags is not None:
conditions += [
- SqlRegisteredModel.tags.any(SqlRegisteredModelTag.tag.contains(tag))
- for tag in filter_tags
+ SqlRegisteredModel.tags.any(SqlRegisteredModelTag.tag.contains(tag)) for tag in filter_tags
]
if filter_str is not None:
conditions.append(SqlRegisteredModel.name.startswith(filter_str))
with self.ManagedSessionMaker() as session:
sql_registered_models = session.query(SqlRegisteredModel).filter(*conditions).all()
return [
- sql_registered_model.to_submarine_entity()
- for sql_registered_model in sql_registered_models
+ sql_registered_model.to_submarine_entity() for sql_registered_model in sql_registered_models
]
def get_registered_model(self, name: str) -> RegisteredModel:
@@ -290,9 +282,7 @@ class SqlAlchemyStore(AbstractStore):
return self._get_sql_registered_model(session, name, True).to_submarine_entity()
@classmethod
- def _get_registered_model_tag(
- cls, session: Session, name: str, tag: str
- ) -> SqlRegisteredModelTag:
+ def _get_registered_model_tag(cls, session: Session, name: str, tag: str) -> SqlRegisteredModelTag:
tags = (
session.query(SqlRegisteredModelTag)
.filter(SqlRegisteredModelTag.name == name, SqlRegisteredModelTag.tag == tag)
@@ -363,7 +353,7 @@ class SqlAlchemyStore(AbstractStore):
def next_version(sql_registered_model: SqlRegisteredModel) -> int:
if sql_registered_model.model_versions:
- return max([m.version for m in sql_registered_model.model_versions]) + 1
+ return max(m.version for m in sql_registered_model.model_versions) + 1
else:
return 1
@@ -420,15 +410,12 @@ class SqlAlchemyStore(AbstractStore):
raise SubmarineException(f"Model Version (name={name}, version={version}) not found.")
elif len(models) > 1:
raise SubmarineException(
- f"Expected only 1 model version with (name={name}, version={version}). Found"
- f" {len(models)}."
+ f"Expected only 1 model version with (name={name}, version={version}). Found {len(models)}."
)
else:
return models[0]
- def update_model_version_description(
- self, name: str, version: int, description: str
- ) -> ModelVersion:
+ def update_model_version_description(self, name: str, version: int, description: str) -> ModelVersion:
"""
Update description associated with the version of model in backend.
:param name: Registered model name.
@@ -502,8 +489,7 @@ class SqlAlchemyStore(AbstractStore):
conditions = [SqlModelVersion.name == name]
if filter_tags is not None:
conditions += [
- SqlModelVersion.tags.any(SqlModelVersionTag.tag.contains(tag))
- for tag in filter_tags
+ SqlModelVersion.tags.any(SqlModelVersionTag.tag.contains(tag)) for tag in filter_tags
]
with self.ManagedSessionMaker() as session:
sql_models = session.query(SqlModelVersion).filter(*conditions).all()
diff --git a/submarine-sdk/pysubmarine/submarine/store/tracking/sqlalchemy_store.py b/submarine-sdk/pysubmarine/submarine/store/tracking/sqlalchemy_store.py
index e7e1e3a6..e13df67b 100644
--- a/submarine-sdk/pysubmarine/submarine/store/tracking/sqlalchemy_store.py
+++ b/submarine-sdk/pysubmarine/submarine/store/tracking/sqlalchemy_store.py
@@ -51,7 +51,7 @@ class SqlAlchemyStore(AbstractStore):
<https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls>`_
for format specifications. Submarine supports the dialects ``mysql``.
"""
- super(SqlAlchemyStore, self).__init__()
+ super().__init__()
self.db_uri = db_uri
self.db_type = extract_db_type_from_uri(db_uri)
self.engine = sqlalchemy.create_engine(db_uri, pool_pre_ping=True)
diff --git a/submarine-sdk/pysubmarine/submarine/tracking/client.py b/submarine-sdk/pysubmarine/submarine/tracking/client.py
index b99540e2..0ad37b88 100644
--- a/submarine-sdk/pysubmarine/submarine/tracking/client.py
+++ b/submarine-sdk/pysubmarine/submarine/tracking/client.py
@@ -31,7 +31,7 @@ from submarine.utils.validation import validate_metric, validate_param
from .constant import AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, S3_ENDPOINT_URL
-class SubmarineClient(object):
+class SubmarineClient:
"""
Client of an submarine Tracking Server that creates and manages experiments and runs.
"""
@@ -181,8 +181,7 @@ class SubmarineClient(object):
if input_dim is None or output_dim is None:
raise Exception(
- "Saving pytorch model needs to provide input and output dimension for"
- " serving."
+ "Saving pytorch model needs to provide input and output dimension for serving."
)
submarine.models.pytorch.save_model(model, model_save_dir, input_dim)
elif model_type == "tensorflow":
@@ -190,7 +189,7 @@ class SubmarineClient(object):
submarine.models.tensorflow.save_model(model, model_save_dir)
else:
- raise Exception("No valid type of model has been matched to {}".format(model_type))
+ raise Exception(f"No valid type of model has been matched to {model_type}")
# Write description file
description["id"] = model_id
diff --git a/submarine-sdk/pysubmarine/submarine/tracking/fluent.py b/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
index a2c8b65e..69c21c0d 100644
--- a/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
+++ b/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
@@ -16,7 +16,6 @@
Internal module implementing the fluent API, allowing management of an active
Submarine run. This module is exposed to users at the top-level :py:mod:`submarine` module.
"""
-from __future__ import print_function
import logging
from datetime import datetime
diff --git a/submarine-sdk/pysubmarine/submarine/tracking/utils.py b/submarine-sdk/pysubmarine/submarine/tracking/utils.py
index 72558358..8c1e1ef1 100644
--- a/submarine-sdk/pysubmarine/submarine/tracking/utils.py
+++ b/submarine-sdk/pysubmarine/submarine/tracking/utils.py
@@ -13,7 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from __future__ import print_function
import json
import os
diff --git a/submarine-sdk/pysubmarine/submarine/utils/__init__.py b/submarine-sdk/pysubmarine/submarine/utils/__init__.py
index 1b0f0454..7b068229 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/__init__.py
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from six.moves import urllib
+import urllib.parse
from submarine.exceptions import SubmarineException
from submarine.utils.db_utils import get_db_uri, set_db_uri
@@ -32,7 +32,7 @@ def extract_db_type_from_uri(db_uri: str):
elif scheme_plus_count == 1:
db_type, _ = scheme.split("+")
else:
- error_msg = "Invalid database URI: '%s'. %s" % (db_uri, "INVALID_DB_URI_MSG")
+ error_msg = f"Invalid database URI: '{db_uri}'. INVALID_DB_URI_MSG"
raise SubmarineException(error_msg)
return db_type
diff --git a/submarine-sdk/pysubmarine/submarine/utils/env.py b/submarine-sdk/pysubmarine/submarine/utils/env.py
index 110a134d..20b5b0ed 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/env.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/env.py
@@ -77,4 +77,4 @@ def get_from_registry(key: str, registry: dict):
if key in registry:
return registry[key]
else:
- raise ValueError("Key {} not supported, available options: {}".format(key, registry.keys()))
+ raise ValueError(f"Key {key} not supported, available options: {registry.keys()}")
diff --git a/submarine-sdk/pysubmarine/submarine/utils/fileio.py b/submarine-sdk/pysubmarine/submarine/utils/fileio.py
index 53a1145a..093698d0 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/fileio.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/fileio.py
@@ -21,9 +21,7 @@ from urllib.parse import urlparse
from pyarrow import fs
-def open_buffered_file_reader(
- uri: str, buffer_size: int = io.DEFAULT_BUFFER_SIZE
-) -> io.BufferedReader:
+def open_buffered_file_reader(uri: str, buffer_size: int = io.DEFAULT_BUFFER_SIZE) -> io.BufferedReader:
try:
input_file = open_input_file(uri)
return io.BufferedReader(input_file, buffer_size=buffer_size)
@@ -32,9 +30,7 @@ def open_buffered_file_reader(
raise e
-def open_buffered_stream_writer(
- uri: str, buffer_size: int = io.DEFAULT_BUFFER_SIZE
-) -> io.BufferedWriter:
+def open_buffered_stream_writer(uri: str, buffer_size: int = io.DEFAULT_BUFFER_SIZE) -> io.BufferedWriter:
try:
output_stream = open_output_stream(uri)
return io.BufferedWriter(output_stream, buffer_size=buffer_size)
diff --git a/submarine-sdk/pysubmarine/submarine/utils/rest_utils.py b/submarine-sdk/pysubmarine/submarine/utils/rest_utils.py
index 5054af02..c05d5767 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/rest_utils.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/rest_utils.py
@@ -65,9 +65,8 @@ def verify_rest_response(response, endpoint: str):
if _can_parse_as_json(response.text):
raise RestException(json.loads(response.text))
else:
- base_msg = "API request to endpoint %s failed with error code %s != 200" % (
- endpoint,
- response.status_code,
+ base_msg = (
+ f"API request to endpoint {endpoint} failed with error code {response.status_code} != 200"
)
- raise SubmarineException("%s. Response body: '%s'" % (base_msg, response.text))
+ raise SubmarineException(f"{base_msg}. Response body: '{response.text}'")
return response
diff --git a/submarine-sdk/pysubmarine/submarine/utils/tf_utils.py b/submarine-sdk/pysubmarine/submarine/utils/tf_utils.py
index 4b79e065..0440535c 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/tf_utils.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/tf_utils.py
@@ -26,12 +26,7 @@ def _get_session_config_from_env_var(params):
tf_config = json.loads(os.environ.get("TF_CONFIG", "{}"))
- if (
- tf_config
- and "task" in tf_config
- and "type" in tf_config["task"]
- and "index" in tf_config["task"]
- ):
+ if tf_config and "task" in tf_config and "type" in tf_config["task"] and "index" in tf_config["task"]:
# Master should only communicate with itself and ps.
if tf_config["task"]["type"] == "master":
return tf.ConfigProto(
@@ -112,9 +107,7 @@ def get_estimator_spec(logit, labels, mode, params):
}
# Provide an estimator spec for `ModeKeys.PREDICT`
if mode == tf.estimator.ModeKeys.PREDICT:
- return tf.estimator.EstimatorSpec(
- mode=mode, predictions=predictions, export_outputs=export_outputs
- )
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, export_outputs=export_outputs)
with tf.name_scope("Loss"):
loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logit, labels=labels))
@@ -137,6 +130,4 @@ def get_estimator_spec(logit, labels, mode, params):
# Provide an estimator spec for `ModeKeys.TRAIN` modes
if mode == tf.estimator.ModeKeys.TRAIN:
- return tf.estimator.EstimatorSpec(
- mode=mode, predictions=predictions, loss=loss, train_op=train_op
- )
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, loss=loss, train_op=train_op)
diff --git a/submarine-sdk/pysubmarine/submarine/utils/tf_utils_v2.py b/submarine-sdk/pysubmarine/submarine/utils/tf_utils_v2.py
index 3c776ef2..80775f89 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/tf_utils_v2.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/tf_utils_v2.py
@@ -26,12 +26,7 @@ def _get_session_config_from_env_var(params):
tf.compat.v1.disable_v2_behavior()
tf_config = json.loads(os.environ.get("TF_CONFIG", "{}"))
- if (
- tf_config
- and "task" in tf_config
- and "type" in tf_config["task"]
- and "index" in tf_config["task"]
- ):
+ if tf_config and "task" in tf_config and "type" in tf_config["task"] and "index" in tf_config["task"]:
# Master should only communicate with itself and ps.
if tf_config["task"]["type"] == "master":
return tf.compat.v1.ConfigProto(
@@ -114,9 +109,7 @@ def get_estimator_spec(logit, labels, mode, params):
}
# Provide an estimator spec for `ModeKeys.PREDICT`
if mode == tf.estimator.ModeKeys.PREDICT:
- return tf.estimator.EstimatorSpec(
- mode=mode, predictions=predictions, export_outputs=export_outputs
- )
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, export_outputs=export_outputs)
with tf.compat.v1.name_scope("Loss"):
loss = tf.reduce_mean(
@@ -141,6 +134,4 @@ def get_estimator_spec(logit, labels, mode, params):
# Provide an estimator spec for `ModeKeys.TRAIN` modes
if mode == tf.estimator.ModeKeys.TRAIN:
- return tf.estimator.EstimatorSpec(
- mode=mode, predictions=predictions, loss=loss, train_op=train_op
- )
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, loss=loss, train_op=train_op)
diff --git a/submarine-sdk/pysubmarine/submarine/utils/validation.py b/submarine-sdk/pysubmarine/submarine/utils/validation.py
index 049a8731..e0671bb4 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/validation.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/validation.py
@@ -54,24 +54,22 @@ def _validate_param_name(name: str):
"""Check that `name` is a valid parameter name and raise an exception if it isn't."""
if not _VALID_PARAM_AND_METRIC_NAMES.match(name):
raise SubmarineException(
- "Invalid parameter name: '%s'. %s" % (name, _BAD_CHARACTERS_MESSAGE),
+ f"Invalid parameter name: '{name}'. {_BAD_CHARACTERS_MESSAGE}",
)
if path_not_unique(name):
- raise SubmarineException(
- "Invalid parameter name: '%s'. %s" % (name, bad_path_message(name))
- )
+ raise SubmarineException(f"Invalid parameter name: '{name}'. {bad_path_message(name)}")
def _validate_metric_name(name: str):
"""Check that `name` is a valid metric name and raise an exception if it isn't."""
if not _VALID_PARAM_AND_METRIC_NAMES.match(name):
raise SubmarineException(
- "Invalid metric name: '%s'. %s" % (name, _BAD_CHARACTERS_MESSAGE),
+ f"Invalid metric name: '{name}'. {_BAD_CHARACTERS_MESSAGE}",
)
if path_not_unique(name):
- raise SubmarineException("Invalid metric name: '%s'. %s" % (name, bad_path_message(name)))
+ raise SubmarineException(f"Invalid metric name: '{name}'. {bad_path_message(name)}")
def _validate_length_limit(entity_name: str, limit: int, value):
@@ -96,14 +94,14 @@ def validate_metric(key, value, timestamp, step) -> None:
if not isinstance(timestamp, datetime):
raise SubmarineException(
- "Got invalid timestamp %s for metric '%s' (value=%s). Timestamp must be a datetime "
- "object." % (timestamp, key, value),
+ f"Got invalid timestamp {timestamp} for metric '{key}' (value={value}). Timestamp must be a"
+ " datetime object."
)
if not isinstance(step, numbers.Number):
raise SubmarineException(
- "Got invalid step %s for metric '%s' (value=%s). Step must be a valid long "
- "(64-bit integer)." % (step, key, value),
+ f"Got invalid step (step) for metric '{key}' (value={value}). Step must be a valid long (64-bit"
+ " integer)."
)
@@ -130,7 +128,7 @@ def validate_tag(tag: str) -> None:
if tag is None or tag == "":
raise SubmarineException("Tag cannot be empty.")
if not _VALID_PARAM_AND_METRIC_NAMES.match(tag):
- raise SubmarineException("Invalid tag name: '%s'. %s" % (tag, _BAD_CHARACTERS_MESSAGE))
+ raise SubmarineException(f"Invalid tag name: '{tag}'. {_BAD_CHARACTERS_MESSAGE}")
def validate_model_name(name: str) -> None:
@@ -149,14 +147,10 @@ def validate_description(description: Optional[str]) -> None:
if not isinstance(description, str) and description is not None:
raise SubmarineException(f"Description must be String or None, but got {type(description)}")
if isinstance(description, str) and len(description) > 5000:
- raise SubmarineException(
- f"Description must less than 5000 words, but got {len(description)}"
- )
+ raise SubmarineException(f"Description must less than 5000 words, but got {len(description)}")
def _validate_db_type_string(db_type):
"""validates db_type parsed from DB URI is supported"""
if db_type not in DATABASE_ENGINES:
- raise SubmarineException(
- f"Invalid database engine: '{db_type}'. '{_UNSUPPORTED_DB_TYPE_MSG}'"
- )
+ raise SubmarineException(f"Invalid database engine: '{db_type}'. '{_UNSUPPORTED_DB_TYPE_MSG}'")
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_config.py b/submarine-sdk/pysubmarine/tests/cli/test_config.py
index 23563d4b..3fdbc077 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_config.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_config.py
@@ -26,8 +26,8 @@ def test_list_config():
_config = loadConfig()
assert result.exit_code == 0
assert "SubmarineCliConfig" in result.output
- assert '"hostname": "{}"'.format(_config.connection.hostname) in result.output
- assert '"port": {}'.format(_config.connection.port) in result.output
+ assert f'"hostname": "{_config.connection.hostname}"' in result.output
+ assert f'"port": {_config.connection.port}' in result.output
def test_init_config():
@@ -36,8 +36,8 @@ def test_init_config():
result = runner.invoke(main.entry_point, ["config", "list"])
_default_config = SubmarineCliConfig()
assert result.exit_code == 0
- assert '"hostname": "{}"'.format(_default_config.connection.hostname) in result.output
- assert '"port": {}'.format(_default_config.connection.port) in result.output
+ assert f'"hostname": "{_default_config.connection.hostname}"' in result.output
+ assert f'"port": {_default_config.connection.port}' in result.output
def test_get_set_experiment():
@@ -47,16 +47,14 @@ def test_get_set_experiment():
result = runner.invoke(main.entry_point, ["config", "get", "connection.hostname"])
assert result.exit_code == 0
_config = loadConfig()
- assert "connection.hostname={}".format(_config.connection.hostname) in result.output
+ assert f"connection.hostname={_config.connection.hostname}" in result.output
- result = runner.invoke(
- main.entry_point, ["config", "set", "connection.hostname", mock_hostname]
- )
+ result = runner.invoke(main.entry_point, ["config", "set", "connection.hostname", mock_hostname])
assert result.exit_code == 0
result = runner.invoke(main.entry_point, ["config", "get", "connection.hostname"])
assert result.exit_code == 0
_config = loadConfig()
- assert "connection.hostname={}".format(mock_hostname) in result.output
+ assert f"connection.hostname={mock_hostname}" in result.output
assert mock_hostname == _config.connection.hostname
initConfig()
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_environment.py b/submarine-sdk/pysubmarine/tests/cli/test_environment.py
index c7144f2d..185cec24 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_environment.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_environment.py
@@ -38,7 +38,7 @@ def test_all_environment_e2e():
# check if cli config is correct for testing
result = runner.invoke(main.entry_point, ["config", "get", "connection.port"])
assert result.exit_code == 0
- assert "connection.port={}".format(8080) in result.output
+ assert "connection.port=8080" in result.output
submarine_client = EnvironmentClient(host="http://localhost:8080")
kernel_spec = KernelSpec(
@@ -66,12 +66,12 @@ def test_all_environment_e2e():
# test get environment
result = runner.invoke(main.entry_point, ["get", "environment", environment_name])
- assert "Environment(name = {} )".format(environment_name) in result.output
+ assert f"Environment(name = {environment_name} )" in result.output
assert environment["environmentSpec"]["name"] in result.output
# test delete environment
result = runner.invoke(main.entry_point, ["delete", "environment", environment_name])
- assert "Environment(name = {} ) deleted".format(environment_name) in result.output
+ assert f"Environment(name = {environment_name} ) deleted" in result.output
# test get environment fail after delete
result = runner.invoke(main.entry_point, ["get", "environment", environment_name])
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_experiment.py b/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
index e6203a9e..b46bd83e 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
@@ -41,7 +41,7 @@ def test_all_experiment_e2e():
# check if cli config is correct for testing
result = runner.invoke(main.entry_point, ["config", "get", "connection.port"])
assert result.exit_code == 0
- assert "connection.port={}".format(8080) in result.output
+ assert "connection.port=8080" in result.output
submarine_client = submarine.ExperimentClient(host="http://localhost:8080")
environment = EnvironmentSpec(image="apache/submarine:tf-dist-mnist-test-1.0")
@@ -82,14 +82,12 @@ def test_all_experiment_e2e():
# test get experiment
result = runner.invoke(main.entry_point, ["get", "experiment", experiment["experimentId"]])
- assert "Experiment(id = {} )".format(experiment["experimentId"]) in result.output
+ assert f"Experiment(id = {experiment['experimentId']} )" in result.output
assert experiment["spec"]["environment"]["image"] in result.output
# test delete experiment (blocking mode)
- result = runner.invoke(
- main.entry_point, ["delete", "experiment", experiment["experimentId"], "--wait"]
- )
- assert "Experiment(id = {} ) deleted".format(experiment["experimentId"]) in result.output
+ result = runner.invoke(main.entry_point, ["delete", "experiment", experiment["experimentId"], "--wait"])
+ assert f"Experiment(id = {experiment['experimentId']} ) deleted" in result.output
# test get experiment fail after delete
result = runner.invoke(main.entry_point, ["get", "experiment", experiment["experimentId"]])
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_notebook.py b/submarine-sdk/pysubmarine/tests/cli/test_notebook.py
index eff73da1..ef0af3a5 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_notebook.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_notebook.py
@@ -40,7 +40,7 @@ def test_all_notbook_e2e():
# check if cli config is correct for testing
result = runner.invoke(main.entry_point, ["config", "get", "connection.port"])
assert result.exit_code == 0
- assert "connection.port={}".format(8080) in result.output
+ assert "connection.port=8080" in result.output
submarine_client = NotebookClient(host="http://localhost:8080")
@@ -48,9 +48,7 @@ def test_all_notbook_e2e():
notebook_meta = NotebookMeta(name="test-nb", namespace="default", owner_id=mock_user_id)
environment = EnvironmentSpec(name="notebook-env")
- notebook_podSpec = NotebookPodSpec(
- env_vars={"TEST_ENV": "test"}, resources="cpu=1,memory=1.0Gi"
- )
+ notebook_podSpec = NotebookPodSpec(env_vars={"TEST_ENV": "test"}, resources="cpu=1,memory=1.0Gi")
notebookSpec = NotebookSpec(meta=notebook_meta, environment=environment, spec=notebook_podSpec)
notebook = submarine_client.create_notebook(notebookSpec)
@@ -68,12 +66,12 @@ def test_all_notbook_e2e():
# test get notebook
result = runner.invoke(main.entry_point, ["get", "notebook", notebookId])
- assert "Notebook(id = {} )".format(notebookId) in result.output
+ assert f"Notebook(id = {notebookId} )" in result.output
assert notebook["spec"]["environment"]["name"] in result.output
# test delete notebook
result = runner.invoke(main.entry_point, ["delete", "notebook", notebookId])
- assert "Notebook(id = {} ) deleted".format(notebookId) in result.output
+ assert f"Notebook(id = {notebookId} ) deleted" in result.output
# test get environment fail after delete
result = runner.invoke(main.entry_point, ["get", "notebook", notebookId])
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_sandbox.py b/submarine-sdk/pysubmarine/tests/cli/test_sandbox.py
index fb8d9ee0..ba953f12 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_sandbox.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_sandbox.py
@@ -24,12 +24,12 @@ def test_start_sandbox():
runner = CliRunner()
result = runner.invoke(main.entry_point, ["sandbox", "start"])
assert result.exit_code == 0
- assert "start sandbox! version={}".format(default_version) in result.output
+ assert f"start sandbox! version={default_version}" in result.output
runner = CliRunner()
result = runner.invoke(main.entry_point, ["sandbox", "start", "-v", mock_version])
assert result.exit_code == 0
- assert "start sandbox! version={}".format(mock_version) in result.output
+ assert f"start sandbox! version={mock_version}" in result.output
def test_delete_sandbox():
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
index e9aff62b..7828a887 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
@@ -19,7 +19,6 @@ import tensorflow as tf
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_run_deepfm(get_model_param):
-
from submarine.ml.tensorflow.model import DeepFM
params = get_model_param
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
index e8e5717c..a4603b96 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
@@ -19,7 +19,6 @@ import tensorflow as tf
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_run_fm(get_model_param):
-
from submarine.ml.tensorflow_v2.model import FM
params = get_model_param
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
index e5398125..19f51882 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
@@ -19,7 +19,6 @@ import tensorflow as tf
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_get_optimizer():
-
from submarine.ml.tensorflow_v2.optimizer import get_optimizer
optimizer_keys = ["adam", "adagrad", "momentum", "ftrl"]
diff --git a/submarine-sdk/pysubmarine/tests/notebook/test_notebook_client.py b/submarine-sdk/pysubmarine/tests/notebook/test_notebook_client.py
index 0e124c18..cecc8909 100644
--- a/submarine-sdk/pysubmarine/tests/notebook/test_notebook_client.py
+++ b/submarine-sdk/pysubmarine/tests/notebook/test_notebook_client.py
@@ -30,9 +30,7 @@ def test_notebook_e2e():
notebook_meta = NotebookMeta(name="test-nb", namespace="default", owner_id=mock_user_id)
environment = EnvironmentSpec(name="notebook-env")
- notebook_podSpec = NotebookPodSpec(
- env_vars={"TEST_ENV": "test"}, resources="cpu=1,memory=1.0Gi"
- )
+ notebook_podSpec = NotebookPodSpec(env_vars={"TEST_ENV": "test"}, resources="cpu=1,memory=1.0Gi")
notebookSpec = NotebookSpec(meta=notebook_meta, environment=environment, spec=notebook_podSpec)
notebook = submarine_client.create_notebook(notebookSpec)
diff --git a/submarine-sdk/pysubmarine/tests/store/model_registry/test_sqlalchemy_store.py b/submarine-sdk/pysubmarine/tests/store/model_registry/test_sqlalchemy_store.py
index 2a80bb9c..50834b5e 100644
--- a/submarine-sdk/pysubmarine/tests/store/model_registry/test_sqlalchemy_store.py
+++ b/submarine-sdk/pysubmarine/tests/store/model_registry/test_sqlalchemy_store.py
@@ -39,9 +39,7 @@ freezegun.configure(default_ignore_list=["threading", "tensorflow"])
@pytest.mark.e2e
class TestSqlAlchemyStore(unittest.TestCase):
def setUp(self):
- submarine.set_db_uri(
- "mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test"
- )
+ submarine.set_db_uri("mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test")
self.db_uri = submarine.get_db_uri()
self.store = SqlAlchemyStore(self.db_uri)
@@ -107,12 +105,8 @@ class TestSqlAlchemyStore(unittest.TestCase):
name = "test_rename_RM"
new_name = "test_rename_RN_new"
rm = self.store.create_registered_model(name)
- self.store.create_model_version(
- name, "model_id_0", "test", "application_1234", "tensorflow"
- )
- self.store.create_model_version(
- name, "model_id_1", "test", "application_1235", "tensorflow"
- )
+ self.store.create_model_version(name, "model_id_0", "test", "application_1234", "tensorflow")
+ self.store.create_model_version(name, "model_id_1", "test", "application_1235", "tensorflow")
mv1d = self.store.get_model_version(name, 1)
mv2d = self.store.get_model_version(name, 2)
self.assertEqual(rm.name, name)
@@ -193,11 +187,9 @@ class TestSqlAlchemyStore(unittest.TestCase):
self.assertEqual(rm2mv1.name, rm2mv1.name)
self.assertEqual(rm2mv1.tags, rm2mv1.tags)
- def _compare_registered_model_names(
- self, results: List[RegisteredModel], rms: List[RegisteredModel]
- ):
- result_names = set([result.name for result in results])
- rm_names = set([rm.name for rm in rms])
+ def _compare_registered_model_names(self, results: List[RegisteredModel], rms: List[RegisteredModel]):
+ result_names = {result.name for result in results}
+ rm_names = {rm.name for rm in rms}
self.assertEqual(result_names, rm_names)
@@ -439,9 +431,7 @@ class TestSqlAlchemyStore(unittest.TestCase):
def test_update_model_version_description(self):
name = "test_update_MV_description"
self.store.create_registered_model(name)
- mv1 = self.store.create_model_version(
- name, "model_id_0", "test", "application_1234", "tensorflow"
- )
+ mv1 = self.store.create_model_version(name, "model_id_0", "test", "application_1234", "tensorflow")
m1d = self.store.get_model_version(mv1.name, mv1.version)
self.assertEqual(m1d.name, name)
self.assertEqual(m1d.version, 1)
@@ -462,12 +452,8 @@ class TestSqlAlchemyStore(unittest.TestCase):
def test_transition_model_version_stage(self):
name = "test_transition_MV_stage"
self.store.create_registered_model(name)
- mv1 = self.store.create_model_version(
- name, "model_id_0", "test", "application_1234", "tensorflow"
- )
- mv2 = self.store.create_model_version(
- name, "model_id_1", "test", "application_1234", "tensorflow"
- )
+ mv1 = self.store.create_model_version(name, "model_id_0", "test", "application_1234", "tensorflow")
+ mv2 = self.store.create_model_version(name, "model_id_1", "test", "application_1234", "tensorflow")
fake_datetime = datetime.strptime("2021-11-11 11:11:11.111000", "%Y-%m-%d %H:%M:%S.%f")
with freeze_time(fake_datetime):
@@ -580,8 +566,8 @@ class TestSqlAlchemyStore(unittest.TestCase):
self.assertEqual(mvd.tags, tags)
def _compare_model_versions(self, results: List[ModelVersion], mms: List[ModelVersion]) -> None:
- result_versions = set([result.version for result in results])
- model_versions = set([mm.version for mm in mms])
+ result_versions = {result.version for result in results}
+ model_versions = {mm.version for mm in mms}
self.assertEqual(result_versions, model_versions)
@@ -593,9 +579,7 @@ class TestSqlAlchemyStore(unittest.TestCase):
self.store.create_registered_model(name2)
tags = ["tag1", "tag2", "tag3"]
models = [
- self.store.create_model_version(
- name1, "model_id_0", "test", "application_1234", "tensorflow"
- ),
+ self.store.create_model_version(name1, "model_id_0", "test", "application_1234", "tensorflow"),
self.store.create_model_version(
name1, "model_id_1", "test", "application_1234", "tensorflow", tags=[tags[0]]
),
@@ -645,9 +629,7 @@ class TestSqlAlchemyStore(unittest.TestCase):
def test_get_model_version_uri(self):
name = "test_get_model_version_uri"
self.store.create_registered_model(name)
- mv = self.store.create_model_version(
- name, "model_id_0", "test", "application_1234", "tensorflow"
- )
+ mv = self.store.create_model_version(name, "model_id_0", "test", "application_1234", "tensorflow")
uri = self.store.get_model_version_uri(mv.name, mv.version)
self.assertEqual(uri, f"s3://submarine/registry/{mv.id}/{mv.name}/{mv.version}")
diff --git a/submarine-sdk/pysubmarine/tests/store/tracking/test_sqlalchemy_store.py b/submarine-sdk/pysubmarine/tests/store/tracking/test_sqlalchemy_store.py
index 31048009..6ee6c22e 100644
--- a/submarine-sdk/pysubmarine/tests/store/tracking/test_sqlalchemy_store.py
+++ b/submarine-sdk/pysubmarine/tests/store/tracking/test_sqlalchemy_store.py
@@ -30,9 +30,7 @@ JOB_ID = "application_123456789"
@pytest.mark.e2e
class TestSqlAlchemyStore(unittest.TestCase):
def setUp(self):
- submarine.set_db_uri(
- "mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test"
- )
+ submarine.set_db_uri("mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test")
self.db_uri = submarine.get_db_uri()
self.store = SqlAlchemyStore(self.db_uri)
# TODO(KUAN-HSUN-LI): use submarine.tracking.fluent to support experiment create
diff --git a/submarine-sdk/pysubmarine/tests/tracking/test_tracking.py b/submarine-sdk/pysubmarine/tests/tracking/test_tracking.py
index d072b5a5..8955eb7f 100644
--- a/submarine-sdk/pysubmarine/tests/tracking/test_tracking.py
+++ b/submarine-sdk/pysubmarine/tests/tracking/test_tracking.py
@@ -37,9 +37,7 @@ MLFLOW_S3_ENDPOINT_URL = "http://localhost:9000"
class TestTracking(unittest.TestCase):
def setUp(self):
environ["JOB_ID"] = JOB_ID
- submarine.set_db_uri(
- "mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test"
- )
+ submarine.set_db_uri("mysql+pymysql://submarine_test:password_test@localhost:3306/submarine_test")
self.db_uri = submarine.get_db_uri()
self.client = SubmarineClient(
db_uri=self.db_uri,
@@ -99,8 +97,8 @@ class TestTracking(unittest.TestCase):
input_arr = tensorflow.random.uniform((1, 5))
model = LinearNNModel()
model(input_arr)
- self.client.save_model(model, "tensorflow", "name_1", REGISTERED_MODEL_NAME)
- self.client.save_model(model, "tensorflow", "name_2", REGISTERED_MODEL_NAME)
+ self.client.save_model(model, "tensorflow", REGISTERED_MODEL_NAME)
+ self.client.save_model(model, "tensorflow", REGISTERED_MODEL_NAME)
# Validate model_versions
model_versions = self.model_registry.list_model_versions(REGISTERED_MODEL_NAME)
assert len(model_versions) == 2
diff --git a/submarine-sdk/pysubmarine/tests/tracking/test_utils.py b/submarine-sdk/pysubmarine/tests/tracking/test_utils.py
index 121691d9..5306cd89 100644
--- a/submarine-sdk/pysubmarine/tests/tracking/test_utils.py
+++ b/submarine-sdk/pysubmarine/tests/tracking/test_utils.py
@@ -14,8 +14,7 @@
# limitations under the License.
import os
-
-import mock
+from unittest import mock
from submarine.store import DEFAULT_SUBMARINE_JDBC_URL
from submarine.store.tracking.sqlalchemy_store import SqlAlchemyStore
diff --git a/submarine-sdk/pysubmarine/tests/tracking/tf_model.py b/submarine-sdk/pysubmarine/tests/tracking/tf_model.py
index 6b598b90..a4f4b2b0 100644
--- a/submarine-sdk/pysubmarine/tests/tracking/tf_model.py
+++ b/submarine-sdk/pysubmarine/tests/tracking/tf_model.py
@@ -19,7 +19,7 @@ import tensorflow as tf
class LinearNNModel(tf.keras.Model):
def __init__(self):
- super(LinearNNModel, self).__init__()
+ super().__init__()
self.dense1 = tf.keras.layers.Dense(1, activation=tf.nn.relu) # One in and one out
def call(self, x):
diff --git a/submarine-sdk/pysubmarine/tests/utils/test_db_utils.py b/submarine-sdk/pysubmarine/tests/utils/test_db_utils.py
index 25650827..446d1ece 100644
--- a/submarine-sdk/pysubmarine/tests/utils/test_db_utils.py
+++ b/submarine-sdk/pysubmarine/tests/utils/test_db_utils.py
@@ -14,8 +14,7 @@
# limitations under the License.
import os
-
-import mock
+from unittest import mock
from submarine.store import DEFAULT_SUBMARINE_JDBC_URL
from submarine.utils import get_db_uri, set_db_uri
diff --git a/submarine-sdk/pysubmarine/tests/utils/test_rest_utils.py b/submarine-sdk/pysubmarine/tests/utils/test_rest_utils.py
index f8da8fa3..c736fc27 100644
--- a/submarine-sdk/pysubmarine/tests/utils/test_rest_utils.py
+++ b/submarine-sdk/pysubmarine/tests/utils/test_rest_utils.py
@@ -14,18 +14,16 @@
# limitations under the License.
import json
+from unittest.mock import Mock, patch
import pytest
-from mock import Mock, patch
from submarine.exceptions import RestException, SubmarineException
from submarine.utils.rest_utils import http_request, verify_rest_response
def test_http_request():
- dummy_json = json.dumps(
- {"result": {"jobId": "job_1234567", "name": "submarine", "identifier": "test"}}
- )
+ dummy_json = json.dumps({"result": {"jobId": "job_1234567", "name": "submarine", "identifier": "test"}})
with patch("requests.request") as mock_requests:
mock_requests.return_value.text = dummy_json
diff --git a/website/docs/devDocs/Dependencies.md b/website/docs/devDocs/Dependencies.md
index 72b2422f..e30938db 100644
--- a/website/docs/devDocs/Dependencies.md
+++ b/website/docs/devDocs/Dependencies.md
@@ -66,8 +66,11 @@ under the License.
| 1.17 (or later) | To be verified |
## Python
+
| Python Version | Support? |
| ------------- |:-------------:|
-| 3.5 (or earlier) | X |
-| 3.6, 3.7 | √ |
-| 3.8 (or later) | To be verified |
+| 3.6 (or earlier) | X |
+| 3.7 | √ |
+| 3.8 | √ |
+| 3.9 | √ |
+| 3.10 | √ |
diff --git a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
index d6e333b9..efb859dd 100644
--- a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
+++ b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
@@ -24,7 +24,7 @@ This is required for developing & testing changes, we recommend installing pysub
in its own conda environment by running the following
```bash
-conda create --name submarine-dev python=3.6
+conda create --name submarine-dev python=3.7
conda activate submarine-dev
# Install auto-format and lints from current checkout
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org