You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2023/10/12 13:27:35 UTC
[dolphinscheduler-sdk-python] branch 4.0.x updated (3a8a4e3 -> 15cf87d)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a change to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
from 3a8a4e3 fix: Upload resource error in 4.0.x
new 6c88a76 fix: missing key file in asdist subcommand (#84)
new 8bc499c chore: Correct some desc for this repos (#85)
new e22b308 release prepare job
new c2af96a Revert "support cache task (#50)"
new ab6a358 doc: Add backward compatibility check before release (#86)
new bf11520 doc: Change release doc description (#89)
new bf6a2ac doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
new 8b50d83 fix: Correct the bad env name of users tenant (#97)
new 46d2836 doc: correct crontab in schedule (#104)
new 77b5845 doc: correct desc in README.md (#105)
new 55683f6 doc: Add how to config alert (#112)
new 15cf87d feat: Add task group to task class (#114)
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dlc.json | 3 +-
.github/workflows/ci.yaml | 2 +-
src/pydolphinscheduler/__init__.py => MANIFEST.in | 9 +++--
README.md | 4 +-
RELEASE.md | 26 ++++++-------
docs/source/concept.rst | 47 ++++++++++++++++++++++-
docs/source/conf.py | 6 ---
docs/source/index.rst | 4 +-
examples/yaml_define/MoreConfiguration.yaml | 1 -
setup.cfg | 1 -
src/pydolphinscheduler/__init__.py | 2 +-
src/pydolphinscheduler/configuration.py | 2 +-
src/pydolphinscheduler/constants.py | 7 ----
src/pydolphinscheduler/core/task.py | 22 +++++------
src/pydolphinscheduler/version_ext | 2 +-
tests/core/test_engine.py | 3 +-
tests/core/test_task.py | 3 +-
17 files changed, 89 insertions(+), 55 deletions(-)
copy src/pydolphinscheduler/__init__.py => MANIFEST.in (87%)
[dolphinscheduler-sdk-python] 10/12: doc: correct desc in README.md (#105)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 77b5845948e0f88017150b5f0054a1ca333a2131
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Aug 24 10:13:03 2023 +0800
doc: correct desc in README.md (#105)
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 62c10d1..45fd5e9 100644
--- a/README.md
+++ b/README.md
@@ -87,7 +87,7 @@ python ./tutorial.py
```
> NOTICE: Since Apache DolphinScheduler's tenant is requests while running command, you have to change
-> tenant value in tutorial.py`. The default value is `tenant_exists`, change it to username exists your host.
+> tenant value in file tutorial.py. The default value is `tenant_exists`, change it to username exists your host.
After that, a new workflow will be created by PyDolphinScheduler, and you can see it in DolphinScheduler web
UI's Project Management page. It will trigger the workflow automatically, so you can see the workflow running
[dolphinscheduler-sdk-python] 02/12: chore: Correct some desc for this repos (#85)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 8bc499c9acea7e9f153f56b677dfab459a83e709
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Tue Mar 28 12:05:33 2023 +0800
chore: Correct some desc for this repos (#85)
---
.github/workflows/ci.yaml | 2 +-
RELEASE.md | 12 ++++++------
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index e76d26f..958f8eb 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -154,7 +154,7 @@ jobs:
if: ${{ github.event_name == 'schedule' || contains(toJSON(github.event.commits.*.message), '[run-it]') }}
timeout-minutes: 30
steps:
- - name: Checkout Dolphinscheduler SDK Python
+ - name: Checkout Dolphinscheduler Python SDK
uses: actions/checkout@v3
with:
path: src
diff --git a/RELEASE.md b/RELEASE.md
index 1c241ba..16ba283 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -82,13 +82,13 @@ cd release/dolphinscheduler && svn add python/"${VERSION}" && svn commit python
Create a new vote in dev@dolphinscheduler.apache.org
```text
-TITLE: [VOTE] Release Apache DolphinScheduler SDK Python <VERSION>
+TITLE: [VOTE] Release Apache DolphinScheduler Python SDK <VERSION>
BODY:
Hello DolphinScheduler Community,
-This is a call for the vote to release Apache DolphinScheduler SDK Python version <VERSION>
+This is a call for the vote to release Apache DolphinScheduler Python SDK version <VERSION>
Release notes: https://github.com/apache/dolphinscheduler-sdk-python/releases/tag/<VERSION>
@@ -146,7 +146,7 @@ Vote result should follow these:
- **Send the closing vote mail to announce the result**. When count the binding and no binding votes, please list the names of voters. An example like this:
```text
- [RESULT][VOTE] Release Apache SkyWalking Python version $VERSION
+ [RESULT][VOTE] Release Apache DolphinScheduler Python version $VERSION
72+ hours passed, we’ve got ($NUMBER) +1 bindings (and ... +1 non-bindings):
@@ -177,15 +177,15 @@ Vote result should follow these:
- Send ANNOUNCE email to `dev@dolphinscheduler.apache.org` and `announce@apache.org`.
```text
- Subject: [ANNOUNCE] Apache DolphinScheduler SDK Python $VERSION Released
+ Subject: [ANNOUNCE] Apache DolphinScheduler Python SDK $VERSION Released
Content:
Hi Community,
- We are glad to announce the release of Apache DolphinScheduler SDK Python $VERSION.
+ We are glad to announce the release of Apache DolphinScheduler Python SDK $VERSION.
- Apache DolphinScheduler SDK Python is an API for Apache DolphinScheduler which allow you definition your workflow by Python code, aka workflow-as-codes.
+ Apache DolphinScheduler Python SDK is an API for Apache DolphinScheduler which allow you definition your workflow by Python code, aka workflow-as-codes.
DolphinScheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
[dolphinscheduler-sdk-python] 08/12: fix: Correct the bad env name of users tenant (#97)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 8b50d83d9161bd7fa6821f4c095f5d1de48298e4
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Jul 20 15:42:18 2023 +0800
fix: Correct the bad env name of users tenant (#97)
fix: #94
---
.dlc.json | 3 ++-
src/pydolphinscheduler/configuration.py | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/.dlc.json b/.dlc.json
index bd24e4c..2e3e67c 100644
--- a/.dlc.json
+++ b/.dlc.json
@@ -20,6 +20,7 @@
"retryCount": 10,
"fallbackRetryDelay": "1000s",
"aliveStatusCodes": [
- 200
+ 200,
+ 0
]
}
diff --git a/src/pydolphinscheduler/configuration.py b/src/pydolphinscheduler/configuration.py
index 9d12afe..91ecba9 100644
--- a/src/pydolphinscheduler/configuration.py
+++ b/src/pydolphinscheduler/configuration.py
@@ -184,7 +184,7 @@ USER_PASSWORD = os.environ.get(
"PYDS_USER_PASSWORD", configs.get("default.user.password")
)
USER_EMAIL = os.environ.get("PYDS_USER_EMAIL", configs.get("default.user.email"))
-USER_TENANT = os.environ.get("PYDS_USER_STATE", configs.get("default.user.tenant"))
+USER_TENANT = os.environ.get("PYDS_USER_TENANT", configs.get("default.user.tenant"))
USER_PHONE = str(os.environ.get("PYDS_USER_PHONE", configs.get("default.user.phone")))
USER_STATE = get_int(
os.environ.get("PYDS_USER_STATE", configs.get("default.user.state"))
[dolphinscheduler-sdk-python] 12/12: feat: Add task group to task class (#114)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 15cf87d4dea55c7d14a69419cb404574d9e93b79
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Oct 12 19:49:40 2023 +0800
feat: Add task group to task class (#114)
```py
extract = Shell(
name="extract",
command="echo 'Some extract command here'",
task_group_id=1,
task_group_priority=123
)
```
fix: #106
---
docs/source/concept.rst | 35 +++++++++++++++++++++++++++++++++++
src/pydolphinscheduler/core/task.py | 9 +++++++++
tests/core/test_engine.py | 2 ++
tests/core/test_task.py | 2 ++
4 files changed, 48 insertions(+)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index 0d25ea5..ab8df5e 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -193,6 +193,41 @@ decide workflow of task. You could set `workflow` in both normal assign or in co
With both `Workflow`_, `Tasks`_ and `Tasks Dependence`_, we could build a workflow with multiple tasks.
+Task Group
+~~~~~~~~~~
+
+A task group can manage and control the maximum number of concurrently running tasks. This is particularly
+useful when you want to limit the simultaneous execution of various task types. For instance, in an ETL
+(Extract, Transform, Load) job where data is extracted from a source database, it's crucial to control the
+parallelism of extract tasks to prevent an excessive number of connections to the source database. This is
+where a task group comes into play. There are two key parameters, ``task_group_id`` and ``task_group_priority``
+that determine the behavior of the task group.
+
+Task group can control the maximum number of tasks running at the same time. It is useful when you don't want
+to run too many type of tasks at the same time. For example when you extract data from source database in ELT
+job, you want to control the parallelism of extract task to avoid too many connections to source database.
+Then task group can help you. There are two major parameters ``task_group_id`` and ``task_group_priority``
+to control the behavior of task group.
+
+* ``task_group_id``: is an integer used to identify the task group. You can set a ``task_group_id`` to
+ restrict the parallelism of tasks. The ``task_group_id`` can be find in the DolphinScheduler web UI. The
+ default value is ``0``, which means there are no restrictions for this task group.
+* ``task_group_priority``: is an integer used to define the priority of the task group. When different tasks
+ share the same ``task_group_id``, the task group's priority comes into play, controlling the order in which
+ they run. Higher values indicate higher priority. The default value is ``0``, which means there's no
+ specific priority for this task group, and tasks will run in the order they were created.
+
+Here's an example in Python:
+
+.. code-block:: python
+
+ extract = Shell(
+ name="extract",
+ command="echo 'Some extract command here'",
+ task_group_id=1,
+ task_group_priority=123
+ )
+
Resource Files
--------------
diff --git a/src/pydolphinscheduler/core/task.py b/src/pydolphinscheduler/core/task.py
index c4a0531..b0f27e4 100644
--- a/src/pydolphinscheduler/core/task.py
+++ b/src/pydolphinscheduler/core/task.py
@@ -91,6 +91,9 @@ class Task(Base):
:param task_priority: default TaskPriority.MEDIUM
:param worker_group: default configuration.WORKFLOW_WORKER_GROUP
:param environment_name: default None
+ :param task_group_id: Identify of task group to restrict the parallelism of tasks instance run, default 0.
+ :param task_group_priority: Priority for same task group to, the higher the value, the higher the
+ priority, default 0.
:param delay_time: deault 0
:param fail_retry_times: default 0
:param fail_retry_interval: default 1
@@ -118,6 +121,8 @@ class Task(Base):
"delay_time",
"fail_retry_times",
"fail_retry_interval",
+ "task_group_id",
+ "task_group_priority",
"timeout_flag",
"timeout_notify_strategy",
"timeout",
@@ -150,6 +155,8 @@ class Task(Base):
task_priority: Optional[str] = TaskPriority.MEDIUM,
worker_group: Optional[str] = configuration.WORKFLOW_WORKER_GROUP,
environment_name: Optional[str] = None,
+ task_group_id: Optional[int] = 0,
+ task_group_priority: Optional[int] = 0,
delay_time: Optional[int] = 0,
fail_retry_times: Optional[int] = 0,
fail_retry_interval: Optional[int] = 1,
@@ -172,6 +179,8 @@ class Task(Base):
self.task_priority = task_priority
self.worker_group = worker_group
self._environment_name = environment_name
+ self.task_group_id = task_group_id
+ self.task_group_priority = task_group_priority
self.fail_retry_times = fail_retry_times
self.fail_retry_interval = fail_retry_interval
self.delay_time = delay_time
diff --git a/tests/core/test_engine.py b/tests/core/test_engine.py
index ba44fad..f7d97df 100644
--- a/tests/core/test_engine.py
+++ b/tests/core/test_engine.py
@@ -108,6 +108,8 @@ def test_property_task_params(mock_resource, mock_code_version, attr, expect):
"version": 1,
"description": None,
"delayTime": 0,
+ "taskGroupId": 0,
+ "taskGroupPriority": 0,
"taskType": "test-engine",
"taskParams": {
"mainClass": "org.apache.examples.mock.Mock",
diff --git a/tests/core/test_task.py b/tests/core/test_task.py
index d7aeb81..9781b6e 100644
--- a/tests/core/test_task.py
+++ b/tests/core/test_task.py
@@ -243,6 +243,8 @@ def test_task_get_define():
"version": version,
"description": None,
"delayTime": 0,
+ "taskGroupId": 0,
+ "taskGroupPriority": 0,
"taskType": task_type,
"taskParams": {
"resourceList": [],
[dolphinscheduler-sdk-python] 11/12: doc: Add how to config alert (#112)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 55683f640011ddb7610b80ca79f07f802dc25b1a
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Oct 12 18:18:07 2023 +0800
doc: Add how to config alert (#112)
---
docs/source/concept.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index a3290e3..0d25ea5 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -123,6 +123,16 @@ Parameter ``execution type`` can be set in
* Via environment variables, configurations file setting, for more detail about those way setting, you can see
you can read :doc:`config` section.
+Alert
+~~~~~
+
+Alert is the way to notify user when workflow instance is success or failed. We can set alert with parameter
+``warning_type`` and ``warning_group_id`` in workflow definition.
+
+* ``warning_type`` represent the type of alert, when workflow instance in those status, it will trigger alert.
+ The value of ``warning_type`` could be one of ``failure``, ``success``, ``all``, ``none``.
+* ``warning_group_id`` represent the group of alert, you can get the group id from DolphinScheduler web UI.
+
Tasks
-----
[dolphinscheduler-sdk-python] 07/12: doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit bf6a2ac8b62bbba6fef70a23dcd2e1261fb7c3f5
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Fri Apr 28 14:49:10 2023 +0800
doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
---
RELEASE.md | 2 +-
docs/source/index.rst | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 27d3dc5..a1695d5 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -24,7 +24,7 @@ but we also have a [PyPi](#release-to-pypi) repository for Python package distri
## Prepare
-* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers)
+* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers), and update support version in `docs/source/index.rst` file.
* Cherry-pick some backward compatibility commit in [4.0.x](https://github.com/apache/dolphinscheduler-sdk-python/tree/4.0.x), the first
commit is 3a8a4e31
* Revert commit Apache dolphinscheduler not release yet, see [this link](https://github.com/apache/dolphinscheduler-sdk-python/pulls?q=is%3Apr+is%3Aclosed+label%3Adep-main-repo)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 89611e3..330b4e6 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -84,7 +84,9 @@ use PyDolphinScheduler above version 4.0.0.
+-----------------------------+----------------------------+---------------------+
| above 3.1.0 and prior 3.1.2 | 3.1.0 | before we seperated |
+-----------------------------+----------------------------+---------------------+
-| above 3.1.2 and prior 3.1.4 | 4.0.0 | |
+| above 3.1.2 and prior 3.1.4 | >=4.0.0, <4.0.3 | |
++-----------------------------+----------------------------+---------------------+
+| 3.1.5 | 4.0.3 | |
+-----------------------------+----------------------------+---------------------+
[dolphinscheduler-sdk-python] 09/12: doc: correct crontab in schedule (#104)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 46d28365d7bdb23f02ee5b263a8d67bb822c7618
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Aug 24 08:54:44 2023 +0800
doc: correct crontab in schedule (#104)
---
README.md | 2 +-
docs/source/concept.rst | 2 +-
docs/source/conf.py | 6 ------
3 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/README.md b/README.md
index 3bbcef8..62c10d1 100644
--- a/README.md
+++ b/README.md
@@ -23,7 +23,7 @@ under the License.
[](https://pypi.org/project/apache-dolphinscheduler/)
[](https://raw.githubusercontent.com/apache/dolphinscheduler-sdk-python/main/LICENSE)
[](https://pypi.org/project/apache-dolphinscheduler/)
-[](https://pepy.tech/project/apache-dolphinscheduler)
+[](https://pepy.tech/project/apache-dolphinscheduler)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index 28a797a..a3290e3 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -63,7 +63,7 @@ asterisks expression, and each of the meaning of position as below
┬ ┬ ┬ ┬ ┬ ┬ ┬
│ │ │ │ │ │ │
│ │ │ │ │ │ └─── year
- │ │ │ │ │ └───── day of week (0 - 7) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
+ │ │ │ │ │ └───── day of week (1 - 7) (1 to 7 are Sunday to Saturday, or use names; 7 is for Sunday, or use `SUN`)
│ │ │ │ └─────── month (1 - 12)
│ │ │ └───────── day of month (1 - 31)
│ │ └─────────── hour (0 - 23)
diff --git a/docs/source/conf.py b/docs/source/conf.py
index e834971..03f675d 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -26,7 +26,6 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
-import base64
import os
import sys
from pathlib import Path
@@ -112,11 +111,6 @@ autodoc_default_options = {
autosectionlabel_prefix_document = True
-# extensions for sphinx_github_changelog, token from Jay Chung with None permission scope. have to encode it
-# due to github will delete token string if it finds in any commit
-token_encode = b"Z2hwXzlhczh1ZG1zYTcxbFpPODZZelQzRTVJZHpLYjNDRzBvZzNEUQ=="
-sphinx_github_changelog_token = base64.b64decode(token_encode).decode()
-
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
[dolphinscheduler-sdk-python] 03/12: release prepare job
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit e22b3085ef880ba916d4d92f3bf862f8bb086caf
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Mon Apr 3 16:35:04 2023 +0800
release prepare job
---
setup.cfg | 1 -
src/pydolphinscheduler/__init__.py | 2 +-
src/pydolphinscheduler/version_ext | 2 +-
3 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/setup.cfg b/setup.cfg
index ad67231..9ee93d7 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -117,7 +117,6 @@ doc =
# Unreleased package have a feature we want(use correct version package for API ref), so we install from
# GitHub directly, see also:
# https://github.com/Holzhaus/sphinx-multiversion/issues/42#issuecomment-1210539786
- sphinx-multiversion @ git+https://github.com/Holzhaus/sphinx-multiversion#egg=sphinx-multiversion
sphinx-github-changelog
dev =
# build
diff --git a/src/pydolphinscheduler/__init__.py b/src/pydolphinscheduler/__init__.py
index cf0fe9f..6012fd9 100644
--- a/src/pydolphinscheduler/__init__.py
+++ b/src/pydolphinscheduler/__init__.py
@@ -17,4 +17,4 @@
"""Init root of pydolphinscheduler."""
-__version__ = "4.1.0-dev"
+__version__ = "4.0.3"
diff --git a/src/pydolphinscheduler/version_ext b/src/pydolphinscheduler/version_ext
index 1983d1a..1e5e3b4 100644
--- a/src/pydolphinscheduler/version_ext
+++ b/src/pydolphinscheduler/version_ext
@@ -1 +1 @@
-dolphinscheduler>=3.2.0
\ No newline at end of file
+dolphinscheduler>=3.1.5, <3.1.7
\ No newline at end of file
[dolphinscheduler-sdk-python] 04/12: Revert "support cache task (#50)"
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit c2af96ad80551f5214f9ad83012799986786bb25
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Mon Apr 3 16:38:24 2023 +0800
Revert "support cache task (#50)"
This reverts commit cf623496449219a3134f82de62c452bc30fe1d22.
---
examples/yaml_define/MoreConfiguration.yaml | 1 -
src/pydolphinscheduler/constants.py | 7 -------
src/pydolphinscheduler/core/task.py | 13 -------------
tests/core/test_engine.py | 1 -
tests/core/test_task.py | 1 -
5 files changed, 23 deletions(-)
diff --git a/examples/yaml_define/MoreConfiguration.yaml b/examples/yaml_define/MoreConfiguration.yaml
index 849ee4b..0cda5ed 100644
--- a/examples/yaml_define/MoreConfiguration.yaml
+++ b/examples/yaml_define/MoreConfiguration.yaml
@@ -36,7 +36,6 @@ tasks:
fail_retry_times: 30
fail_retry_interval: 5
timeout: 60
- is_cache: true
input_params:
value_VARCHAR: "abc"
value_INTEGER: 123
diff --git a/src/pydolphinscheduler/constants.py b/src/pydolphinscheduler/constants.py
index b28d3b6..1a5e11f 100644
--- a/src/pydolphinscheduler/constants.py
+++ b/src/pydolphinscheduler/constants.py
@@ -35,13 +35,6 @@ class TaskFlag(str):
NO = "NO"
-class IsCache(str):
- """Constants for Cache."""
-
- YES = "YES"
- NO = "NO"
-
-
class TaskTimeoutFlag(str):
"""Constants for task timeout flag."""
diff --git a/src/pydolphinscheduler/core/task.py b/src/pydolphinscheduler/core/task.py
index 3341040..c4a0531 100644
--- a/src/pydolphinscheduler/core/task.py
+++ b/src/pydolphinscheduler/core/task.py
@@ -26,7 +26,6 @@ from typing import Dict, List, Optional, Sequence, Set, Tuple, Union
from pydolphinscheduler import configuration
from pydolphinscheduler.constants import (
Delimiter,
- IsCache,
ResourceKey,
Symbol,
TaskFlag,
@@ -101,7 +100,6 @@ class Task(Base):
:param wait_start_timeout: default None
:param condition_result: default None,
:param resource_plugin: default None
- :param is_cache: default False
:param input_params: default None, input parameters, {param_name: param_value}
:param output_params: default None, input parameters, {param_name: param_value}
"""
@@ -123,7 +121,6 @@ class Task(Base):
"timeout_flag",
"timeout_notify_strategy",
"timeout",
- "is_cache",
}
# task default attribute will into `task_params` property
@@ -164,7 +161,6 @@ class Task(Base):
wait_start_timeout: Optional[Dict] = None,
condition_result: Optional[Dict] = None,
resource_plugin: Optional[ResourcePlugin] = None,
- is_cache: Optional[bool] = False,
input_params: Optional[Dict] = None,
output_params: Optional[Dict] = None,
*args,
@@ -173,7 +169,6 @@ class Task(Base):
super().__init__(name, description)
self.task_type = task_type
self.flag = flag
- self._is_cache = is_cache
self.task_priority = task_priority
self.worker_group = worker_group
self._environment_name = environment_name
@@ -251,14 +246,6 @@ class Task(Base):
"""Whether the timeout attribute is being set or not."""
return TaskTimeoutFlag.ON if self._timeout else TaskTimeoutFlag.OFF
- @property
- def is_cache(self) -> str:
- """Whether the cache is being set or not."""
- if isinstance(self._is_cache, bool):
- return IsCache.YES if self._is_cache else IsCache.NO
- else:
- raise PyDSParamException("is_cache must be a bool")
-
@property
def resource_list(self) -> List[Dict[str, Resource]]:
"""Get task define attribute `resource_list`."""
diff --git a/tests/core/test_engine.py b/tests/core/test_engine.py
index 90a14d0..ba44fad 100644
--- a/tests/core/test_engine.py
+++ b/tests/core/test_engine.py
@@ -122,7 +122,6 @@ def test_property_task_params(mock_resource, mock_code_version, attr, expect):
"waitStartTimeout": {},
},
"flag": "YES",
- "isCache": "NO",
"taskPriority": "MEDIUM",
"workerGroup": "default",
"environmentCode": None,
diff --git a/tests/core/test_task.py b/tests/core/test_task.py
index d34b6d2..d7aeb81 100644
--- a/tests/core/test_task.py
+++ b/tests/core/test_task.py
@@ -252,7 +252,6 @@ def test_task_get_define():
"waitStartTimeout": {},
},
"flag": "YES",
- "isCache": "NO",
"taskPriority": "MEDIUM",
"workerGroup": "default",
"environmentCode": None,
[dolphinscheduler-sdk-python] 05/12: doc: Add backward compatibility check before release (#86)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit ab6a3582527f1d722b03afd7ff96b389512d3fe3
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Mon Apr 3 20:10:47 2023 +0800
doc: Add backward compatibility check before release (#86)
---
RELEASE.md | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 16ba283..2f564f6 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -24,13 +24,15 @@ but we also have a [PyPi](#release-to-pypi) repository for Python package distri
## Prepare
-* Remove `sphinx-multiversion` dependency in `setup.py`, we still can not fix this issue
- [Distribute tarball and wheel error with direct dependency](https://github.com/apache/dolphinscheduler/issues/12238)
* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers)
-* Run all test locally, `tox -e local-ci && tox -e local-integrate-test`, after you start dolphinscheduler to
- pass `local-integrate-test`
+* Cherry-pick some backward compatibility commit in [4.0.x](https://github.com/apache/dolphinscheduler-sdk-python/tree/4.0.x), the first
+ commit is 3a8a4e31
* Revert commit Apache dolphinscheduler not release yet, see [this link](https://github.com/apache/dolphinscheduler-sdk-python/pulls?q=is%3Apr+is%3Aclosed+label%3Adep-main-repo)
add check whether apache/dolphinscheduler commit is released, for some commit not release should revert the change
+* Run all test locally, `tox -e local-ci && tox -e local-integrate-test`, after you start dolphinscheduler to
+ pass `local-integrate-test`
+* Remove `sphinx-multiversion` dependency in `setup.cfg`, we still can not fix this issue
+ [Distribute tarball and wheel error with direct dependency](https://github.com/apache/dolphinscheduler/issues/12238)
## Build and Sign Package
[dolphinscheduler-sdk-python] 01/12: fix: missing key file in asdist subcommand (#84)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 6c88a7673650bb725fc5f681982bbb019cce7ca3
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Tue Mar 28 09:59:40 2023 +0800
fix: missing key file in asdist subcommand (#84)
Currently, missing license and notice in
python setup.py asdist command
---
MANIFEST.in | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..bd3be26
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include LICENSE
+include NOTICE
+include README.md
+include CONTRIBUTING.md
+include RELEASE.md
+include UPDATING.md
[dolphinscheduler-sdk-python] 06/12: doc: Change release doc description (#89)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.x
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit bf11520d8ddcee816f04da4181a83b0d7c70ca5e
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Fri Apr 14 14:23:25 2023 +0800
doc: Change release doc description (#89)
---
RELEASE.md | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 2f564f6..27d3dc5 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -187,13 +187,11 @@ Vote result should follow these:
We are glad to announce the release of Apache DolphinScheduler Python SDK $VERSION.
- Apache DolphinScheduler Python SDK is an API for Apache DolphinScheduler which allow you definition your workflow by Python code, aka workflow-as-codes.
+ Apache DolphinScheduler Python SDK is an API for Apache DolphinScheduler which allows you to define your workflow by Python code, aka workflow-as-codes.
DolphinScheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
- Download Links: https://dolphinscheduler.apache.org/#/en-us/download
-
Release Notes: https://github.com/apache/dolphinscheduler-sdk-python/releases/tag/$VERSION
DolphinScheduler Resources: