You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2023/10/12 13:12:26 UTC
[dolphinscheduler-sdk-python] branch 4.0.4-prepare updated (aec75a7 -> 7413035)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a change to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
omit aec75a7 fix: Upload resource error in 4.0.x
new 75f458d doc: Add backward compatibility check before release (#86)
new c01b66d doc: Change release doc description (#89)
new bdfc36a doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
new e0d17bc fix: Correct the bad env name of users tenant (#97)
new dc2117a doc: correct crontab in schedule (#104)
new d301e22 doc: correct desc in README.md (#105)
new 5204011 doc: Add how to config alert (#112)
new 7413035 feat: Add task group to task class (#114)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aec75a7)
\
N -- N -- N refs/heads/4.0.4-prepare (7413035)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dlc.json | 3 ++-
README.md | 4 +--
RELEASE.md | 16 +++++------
docs/source/concept.rst | 47 ++++++++++++++++++++++++++++++++-
docs/source/conf.py | 6 -----
docs/source/index.rst | 4 ++-
src/pydolphinscheduler/configuration.py | 2 +-
src/pydolphinscheduler/core/resource.py | 1 -
src/pydolphinscheduler/core/task.py | 9 +++++++
src/pydolphinscheduler/java_gateway.py | 8 ++----
tests/core/test_engine.py | 2 ++
tests/core/test_task.py | 2 ++
12 files changed, 77 insertions(+), 27 deletions(-)
[dolphinscheduler-sdk-python] 07/08: doc: Add how to config alert (#112)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 52040113363f17c5fecc6dadce405973e4259a80
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Oct 12 18:18:07 2023 +0800
doc: Add how to config alert (#112)
---
docs/source/concept.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index a3290e3..0d25ea5 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -123,6 +123,16 @@ Parameter ``execution type`` can be set in
* Via environment variables, configurations file setting, for more detail about those way setting, you can see
you can read :doc:`config` section.
+Alert
+~~~~~
+
+Alert is the way to notify user when workflow instance is success or failed. We can set alert with parameter
+``warning_type`` and ``warning_group_id`` in workflow definition.
+
+* ``warning_type`` represent the type of alert, when workflow instance in those status, it will trigger alert.
+ The value of ``warning_type`` could be one of ``failure``, ``success``, ``all``, ``none``.
+* ``warning_group_id`` represent the group of alert, you can get the group id from DolphinScheduler web UI.
+
Tasks
-----
[dolphinscheduler-sdk-python] 01/08: doc: Add backward compatibility check before release (#86)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 75f458dc4e9f5e05f20c0398327ffe8135b819c0
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Mon Apr 3 20:10:47 2023 +0800
doc: Add backward compatibility check before release (#86)
---
RELEASE.md | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 16ba283..2f564f6 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -24,13 +24,15 @@ but we also have a [PyPi](#release-to-pypi) repository for Python package distri
## Prepare
-* Remove `sphinx-multiversion` dependency in `setup.py`, we still can not fix this issue
- [Distribute tarball and wheel error with direct dependency](https://github.com/apache/dolphinscheduler/issues/12238)
* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers)
-* Run all test locally, `tox -e local-ci && tox -e local-integrate-test`, after you start dolphinscheduler to
- pass `local-integrate-test`
+* Cherry-pick some backward compatibility commit in [4.0.x](https://github.com/apache/dolphinscheduler-sdk-python/tree/4.0.x), the first
+ commit is 3a8a4e31
* Revert commit Apache dolphinscheduler not release yet, see [this link](https://github.com/apache/dolphinscheduler-sdk-python/pulls?q=is%3Apr+is%3Aclosed+label%3Adep-main-repo)
add check whether apache/dolphinscheduler commit is released, for some commit not release should revert the change
+* Run all test locally, `tox -e local-ci && tox -e local-integrate-test`, after you start dolphinscheduler to
+ pass `local-integrate-test`
+* Remove `sphinx-multiversion` dependency in `setup.cfg`, we still can not fix this issue
+ [Distribute tarball and wheel error with direct dependency](https://github.com/apache/dolphinscheduler/issues/12238)
## Build and Sign Package
[dolphinscheduler-sdk-python] 05/08: doc: correct crontab in schedule (#104)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit dc2117a2939554134594c526d7468ca9c20c2ee5
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Aug 24 08:54:44 2023 +0800
doc: correct crontab in schedule (#104)
---
README.md | 2 +-
docs/source/concept.rst | 2 +-
docs/source/conf.py | 6 ------
3 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/README.md b/README.md
index 3bbcef8..62c10d1 100644
--- a/README.md
+++ b/README.md
@@ -23,7 +23,7 @@ under the License.
[](https://pypi.org/project/apache-dolphinscheduler/)
[](https://raw.githubusercontent.com/apache/dolphinscheduler-sdk-python/main/LICENSE)
[](https://pypi.org/project/apache-dolphinscheduler/)
-[](https://pepy.tech/project/apache-dolphinscheduler)
+[](https://pepy.tech/project/apache-dolphinscheduler)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index 28a797a..a3290e3 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -63,7 +63,7 @@ asterisks expression, and each of the meaning of position as below
┬ ┬ ┬ ┬ ┬ ┬ ┬
│ │ │ │ │ │ │
│ │ │ │ │ │ └─── year
- │ │ │ │ │ └───── day of week (0 - 7) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
+ │ │ │ │ │ └───── day of week (1 - 7) (1 to 7 are Sunday to Saturday, or use names; 7 is for Sunday, or use `SUN`)
│ │ │ │ └─────── month (1 - 12)
│ │ │ └───────── day of month (1 - 31)
│ │ └─────────── hour (0 - 23)
diff --git a/docs/source/conf.py b/docs/source/conf.py
index e834971..03f675d 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -26,7 +26,6 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
-import base64
import os
import sys
from pathlib import Path
@@ -112,11 +111,6 @@ autodoc_default_options = {
autosectionlabel_prefix_document = True
-# extensions for sphinx_github_changelog, token from Jay Chung with None permission scope. have to encode it
-# due to github will delete token string if it finds in any commit
-token_encode = b"Z2hwXzlhczh1ZG1zYTcxbFpPODZZelQzRTVJZHpLYjNDRzBvZzNEUQ=="
-sphinx_github_changelog_token = base64.b64decode(token_encode).decode()
-
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
[dolphinscheduler-sdk-python] 04/08: fix: Correct the bad env name of users tenant (#97)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit e0d17bce87183b2e3a5c7c8448f81459e5ac609e
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Jul 20 15:42:18 2023 +0800
fix: Correct the bad env name of users tenant (#97)
fix: #94
---
.dlc.json | 3 ++-
src/pydolphinscheduler/configuration.py | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/.dlc.json b/.dlc.json
index bd24e4c..2e3e67c 100644
--- a/.dlc.json
+++ b/.dlc.json
@@ -20,6 +20,7 @@
"retryCount": 10,
"fallbackRetryDelay": "1000s",
"aliveStatusCodes": [
- 200
+ 200,
+ 0
]
}
diff --git a/src/pydolphinscheduler/configuration.py b/src/pydolphinscheduler/configuration.py
index 9d12afe..91ecba9 100644
--- a/src/pydolphinscheduler/configuration.py
+++ b/src/pydolphinscheduler/configuration.py
@@ -184,7 +184,7 @@ USER_PASSWORD = os.environ.get(
"PYDS_USER_PASSWORD", configs.get("default.user.password")
)
USER_EMAIL = os.environ.get("PYDS_USER_EMAIL", configs.get("default.user.email"))
-USER_TENANT = os.environ.get("PYDS_USER_STATE", configs.get("default.user.tenant"))
+USER_TENANT = os.environ.get("PYDS_USER_TENANT", configs.get("default.user.tenant"))
USER_PHONE = str(os.environ.get("PYDS_USER_PHONE", configs.get("default.user.phone")))
USER_STATE = get_int(
os.environ.get("PYDS_USER_STATE", configs.get("default.user.state"))
[dolphinscheduler-sdk-python] 08/08: feat: Add task group to task class (#114)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit 741303556516571cbbc75b5fb09e89efa3b160d4
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Oct 12 19:49:40 2023 +0800
feat: Add task group to task class (#114)
```py
extract = Shell(
name="extract",
command="echo 'Some extract command here'",
task_group_id=1,
task_group_priority=123
)
```
fix: #106
---
docs/source/concept.rst | 35 +++++++++++++++++++++++++++++++++++
src/pydolphinscheduler/core/task.py | 9 +++++++++
tests/core/test_engine.py | 2 ++
tests/core/test_task.py | 2 ++
4 files changed, 48 insertions(+)
diff --git a/docs/source/concept.rst b/docs/source/concept.rst
index 0d25ea5..ab8df5e 100644
--- a/docs/source/concept.rst
+++ b/docs/source/concept.rst
@@ -193,6 +193,41 @@ decide workflow of task. You could set `workflow` in both normal assign or in co
With both `Workflow`_, `Tasks`_ and `Tasks Dependence`_, we could build a workflow with multiple tasks.
+Task Group
+~~~~~~~~~~
+
+A task group can manage and control the maximum number of concurrently running tasks. This is particularly
+useful when you want to limit the simultaneous execution of various task types. For instance, in an ETL
+(Extract, Transform, Load) job where data is extracted from a source database, it's crucial to control the
+parallelism of extract tasks to prevent an excessive number of connections to the source database. This is
+where a task group comes into play. There are two key parameters, ``task_group_id`` and ``task_group_priority``
+that determine the behavior of the task group.
+
+Task group can control the maximum number of tasks running at the same time. It is useful when you don't want
+to run too many type of tasks at the same time. For example when you extract data from source database in ELT
+job, you want to control the parallelism of extract task to avoid too many connections to source database.
+Then task group can help you. There are two major parameters ``task_group_id`` and ``task_group_priority``
+to control the behavior of task group.
+
+* ``task_group_id``: is an integer used to identify the task group. You can set a ``task_group_id`` to
+ restrict the parallelism of tasks. The ``task_group_id`` can be find in the DolphinScheduler web UI. The
+ default value is ``0``, which means there are no restrictions for this task group.
+* ``task_group_priority``: is an integer used to define the priority of the task group. When different tasks
+ share the same ``task_group_id``, the task group's priority comes into play, controlling the order in which
+ they run. Higher values indicate higher priority. The default value is ``0``, which means there's no
+ specific priority for this task group, and tasks will run in the order they were created.
+
+Here's an example in Python:
+
+.. code-block:: python
+
+ extract = Shell(
+ name="extract",
+ command="echo 'Some extract command here'",
+ task_group_id=1,
+ task_group_priority=123
+ )
+
Resource Files
--------------
diff --git a/src/pydolphinscheduler/core/task.py b/src/pydolphinscheduler/core/task.py
index c4a0531..b0f27e4 100644
--- a/src/pydolphinscheduler/core/task.py
+++ b/src/pydolphinscheduler/core/task.py
@@ -91,6 +91,9 @@ class Task(Base):
:param task_priority: default TaskPriority.MEDIUM
:param worker_group: default configuration.WORKFLOW_WORKER_GROUP
:param environment_name: default None
+ :param task_group_id: Identify of task group to restrict the parallelism of tasks instance run, default 0.
+ :param task_group_priority: Priority for same task group to, the higher the value, the higher the
+ priority, default 0.
:param delay_time: deault 0
:param fail_retry_times: default 0
:param fail_retry_interval: default 1
@@ -118,6 +121,8 @@ class Task(Base):
"delay_time",
"fail_retry_times",
"fail_retry_interval",
+ "task_group_id",
+ "task_group_priority",
"timeout_flag",
"timeout_notify_strategy",
"timeout",
@@ -150,6 +155,8 @@ class Task(Base):
task_priority: Optional[str] = TaskPriority.MEDIUM,
worker_group: Optional[str] = configuration.WORKFLOW_WORKER_GROUP,
environment_name: Optional[str] = None,
+ task_group_id: Optional[int] = 0,
+ task_group_priority: Optional[int] = 0,
delay_time: Optional[int] = 0,
fail_retry_times: Optional[int] = 0,
fail_retry_interval: Optional[int] = 1,
@@ -172,6 +179,8 @@ class Task(Base):
self.task_priority = task_priority
self.worker_group = worker_group
self._environment_name = environment_name
+ self.task_group_id = task_group_id
+ self.task_group_priority = task_group_priority
self.fail_retry_times = fail_retry_times
self.fail_retry_interval = fail_retry_interval
self.delay_time = delay_time
diff --git a/tests/core/test_engine.py b/tests/core/test_engine.py
index ba44fad..f7d97df 100644
--- a/tests/core/test_engine.py
+++ b/tests/core/test_engine.py
@@ -108,6 +108,8 @@ def test_property_task_params(mock_resource, mock_code_version, attr, expect):
"version": 1,
"description": None,
"delayTime": 0,
+ "taskGroupId": 0,
+ "taskGroupPriority": 0,
"taskType": "test-engine",
"taskParams": {
"mainClass": "org.apache.examples.mock.Mock",
diff --git a/tests/core/test_task.py b/tests/core/test_task.py
index d7aeb81..9781b6e 100644
--- a/tests/core/test_task.py
+++ b/tests/core/test_task.py
@@ -243,6 +243,8 @@ def test_task_get_define():
"version": version,
"description": None,
"delayTime": 0,
+ "taskGroupId": 0,
+ "taskGroupPriority": 0,
"taskType": task_type,
"taskParams": {
"resourceList": [],
[dolphinscheduler-sdk-python] 03/08: doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit bdfc36a5edd93bdd0bb654c85ca29a08f70e2b5d
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Fri Apr 28 14:49:10 2023 +0800
doc: Correct support version between pydolphinscheduler and dolphinscheduler (#92)
---
RELEASE.md | 2 +-
docs/source/index.rst | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 27d3dc5..a1695d5 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -24,7 +24,7 @@ but we also have a [PyPi](#release-to-pypi) repository for Python package distri
## Prepare
-* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers)
+* Change `version_ext` about the dolphinscheduler version current support, the syntax is respect [pep-044](https://peps.python.org/pep-0440/#version-specifiers), and update support version in `docs/source/index.rst` file.
* Cherry-pick some backward compatibility commit in [4.0.x](https://github.com/apache/dolphinscheduler-sdk-python/tree/4.0.x), the first
commit is 3a8a4e31
* Revert commit Apache dolphinscheduler not release yet, see [this link](https://github.com/apache/dolphinscheduler-sdk-python/pulls?q=is%3Apr+is%3Aclosed+label%3Adep-main-repo)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 89611e3..330b4e6 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -84,7 +84,9 @@ use PyDolphinScheduler above version 4.0.0.
+-----------------------------+----------------------------+---------------------+
| above 3.1.0 and prior 3.1.2 | 3.1.0 | before we seperated |
+-----------------------------+----------------------------+---------------------+
-| above 3.1.2 and prior 3.1.4 | 4.0.0 | |
+| above 3.1.2 and prior 3.1.4 | >=4.0.0, <4.0.3 | |
++-----------------------------+----------------------------+---------------------+
+| 3.1.5 | 4.0.3 | |
+-----------------------------+----------------------------+---------------------+
[dolphinscheduler-sdk-python] 02/08: doc: Change release doc description (#89)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit c01b66d1e70f6191eedcf2559844056509ac0686
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Fri Apr 14 14:23:25 2023 +0800
doc: Change release doc description (#89)
---
RELEASE.md | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/RELEASE.md b/RELEASE.md
index 2f564f6..27d3dc5 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -187,13 +187,11 @@ Vote result should follow these:
We are glad to announce the release of Apache DolphinScheduler Python SDK $VERSION.
- Apache DolphinScheduler Python SDK is an API for Apache DolphinScheduler which allow you definition your workflow by Python code, aka workflow-as-codes.
+ Apache DolphinScheduler Python SDK is an API for Apache DolphinScheduler which allows you to define your workflow by Python code, aka workflow-as-codes.
DolphinScheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
- Download Links: https://dolphinscheduler.apache.org/#/en-us/download
-
Release Notes: https://github.com/apache/dolphinscheduler-sdk-python/releases/tag/$VERSION
DolphinScheduler Resources:
[dolphinscheduler-sdk-python] 06/08: doc: correct desc in README.md (#105)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 4.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
commit d301e22b6f69033648f0b5d5ebcd1d7b970d83ef
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Thu Aug 24 10:13:03 2023 +0800
doc: correct desc in README.md (#105)
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 62c10d1..45fd5e9 100644
--- a/README.md
+++ b/README.md
@@ -87,7 +87,7 @@ python ./tutorial.py
```
> NOTICE: Since Apache DolphinScheduler's tenant is requests while running command, you have to change
-> tenant value in tutorial.py`. The default value is `tenant_exists`, change it to username exists your host.
+> tenant value in file tutorial.py. The default value is `tenant_exists`, change it to username exists your host.
After that, a new workflow will be created by PyDolphinScheduler, and you can see it in DolphinScheduler web
UI's Project Management page. It will trigger the workflow automatically, so you can see the workflow running