You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@dolphinscheduler.apache.org by Jay Chung <zh...@apache.org> on 2023/06/03 12:01:36 UTC

[DISCUSS] Version 3.2.0 release and help wanted for testing

Hi all,

I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
release in July 2023 or August 2023.

Actually, I already created a new branch named `3.2.0-prepare` for
it[1], including all features merged to branch dev after we release
3.1.0(date Sep 28, 2022), including many interesting functional
additions and code base refactoring.

Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
to Dolphinscheduler or want to contribute to the latest version is
welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
start the standalone server, or build tarball for cluster mode test
based on your wish. In this version, we need to test the core function
of dolphinscheduler like CURD of project, workflow, task and execute
them. we allow need to test the resource center carefully because it
changes a lot in this version.

You can report dolpinscheduler issue[2] if you find any bugs. We have
a specific version for this incoming version name `3.2.0-prepare` in
the bugs reported template.

And feel free to submit PR to fix bugs, whatever is reported by others or you

Some of the functional features list for 3.2.0:

* User friendly
    * UI only: More hyperlinks jump between resource, adjustable column width
    * Default tenant: Out of box run workflow by default tenant
    * Run or rerun task in task instance
    * Human readable json export result
* Cloud native
    * Kubernetes Real-time log and custom label support
    * Support KEDA autoscaler worker
    * Support Terraform deploy to AWS
    * Multi arch for zookeper and postgresql(all services are supported now)
* Resource center
    * Refactor resource center, we use provider API to list and manage
resource now
    * Allow resource re-upload now
* Datasource
    * New datasource:
snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
Athena
    * Support access key for redshift datasource
* Tasks
    * New tasks: azure datafactory、apache
linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
    * Improve for exists: Add presto for datax, parameter passing for
python and http, sql task multiple sql delimiter, dependent depend
itself
    * Introduce task cache for MLops or other long-run tasks
* Parameter
    * Add new rule of build-in function
    * Introduce file type parameter
    * Startup parameters have the highest priority
* Logs: Introduce remote logs mechanism and support Google Cloud
Storage、S3、OSS up to now
* API:
    * Introduce some new restful API(experimental), including
taskInstance、workflow state、workflowInstance、workflow and
schedule、task relation
    * Can get task instance id when triggering workflow via API
    * Registry: Introduce etcd, jdbc supported

[1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
[2]: https://github.com/apache/dolphinscheduler/issues/new/choose

-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Thanks eye, It is one solution to do that, but we should consider
reducing the number of duplicate libs first instead of making them get
together

On Mon, Sep 11, 2023 at 10:17 AM eye gu <gu...@gmail.com> wrote:
>
> Just merge the dependent third-party packages into the same directory, and
> each service directory is still separate. Just modify the directory
> specified in the startup script for dependencies
>
> Jay Chung <zh...@gmail.com> 于2023年9月11日周一 09:49写道：
>
> > Not yet, but we should not merge them into one single directory, cause
> > we take time to separate our services into multiple directories for
> > easy access and use
> >
> > On Sun, Sep 10, 2023 at 12:15 PM 顾泽民 <gu...@gmail.com> wrote:
> > >
> > > I have an idea, can we merge libs from a single directory into one
> > > directory?
> > >
> > > Jay Chung <zh...@gmail.com> 于2023年9月7日周四 17:14写道：
> > >
> > > > Hi guys,
> > > >
> > > > !!! We need some hero for the tarball size of dolphinscheduler. !!!
> > > > ASF infra team disagrees with the Jira ticket for bumping our SVN
> > > > limit[1], we need to find some way to make our binary tarball smaller
> > > > than 950MB. and the current size of our tarball is almost 1.2GB which
> > > > means we should reduce at least 278MB for the tarball.
> > > >
> > > > We have a DSIP to separate and release plugins in the future[2], but
> > > > it is still WIP, maybe we have to find another way to reduce the size
> > > > of binary tarball
> > > >
> > > > Please let me know if you have any good ideas about our tarball size,
> > > > thanks a lot
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/INFRA-24944
> > > > [2]: https://github.com/apache/dolphinscheduler/issues/1252
> > > >
> > > > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zh...@gmail.com>
> > > > wrote:
> > > > >
> > > > > And if you find some bugfix still WIP and nearly to merge, please let
> > > > > me know in this thread, maybe we could add them to 3.2.0 too, cause
> > we
> > > > > block by SVN limit now
> > > > >
> > > > > Some of our community found out about two of them, and thank
> > Gallardot
> > > > > for bringing those up
> > > > >
> > > > > https://github.com/apache/dolphinscheduler/pull/14328
> > > > > https://github.com/apache/dolphinscheduler/pull/14829
> > > > >
> > > > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > FYI
> > > > > >
> > > > > > I use `gh` cli to batch bump all issues and prs with milestone
> > `3.2.0`
> > > > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> > > > > > (especially bug). so please take care of the bug with milestone
> > > > > > `3.3.0` and still open since Aug 31, 2023
> > > > > >
> > > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <
> > zhongjiajie955@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Version 3.2.0 is nearly close and released during those days,
> > but we
> > > > > > > can not upload it to Apache SVN because too big for our tarball,
> > We
> > > > > > > created a Jira ticket to bump the svn limit for dolphinscheduler
> > and
> > > > > > > it is still in discussion. I will follow up when it has a further
> > > > > > > update
> > > > > > >
> > > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <
> > zhongjiajie955@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I had cherry-picked some recent commits to the branch
> > > > 3.2.0-prepare,
> > > > > > > > anyone who interested in checking 3.2.0 features could clone
> > the
> > > > > > > > latest branch to have a look
> > > > > > > >
> > > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <
> > > > zhongjiajie955@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > I submit new PR[1] to remove test datasource in version
> > 3.2.0,
> > > > in [2]
> > > > > > > > > we introduce the test task and test datasource. It is a good
> > > > feature
> > > > > > > > > to allow user run in test environment.
> > > > > > > > >
> > > > > > > > > But I find out we have to create test datasource before
> > create
> > > > the
> > > > > > > > > prod one. I think it make user confuse especial some one who
> > do
> > > > not
> > > > > > > > > use test environment. So I think we should remove it in
> > version
> > > > 3.2.0.
> > > > > > > > >
> > > > > > > > > If you have different idea, please let me know in this mail
> > > > thread
> > > > > > > > >
> > > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > > > > > > >
> > > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <
> > > > zhongjiajie955@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Community,
> > > > > > > > > >
> > > > > > > > > > I added a checklist for the version 3.2.0 release,
> > everyone who
> > > > > > > > > > interested in the incoming version can participate in our
> > > > test, the
> > > > > > > > > > detail checklist issue is in
> > > > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <
> > > > zhuang.kerwin@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks Jay.
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to the 3.2.0 release.
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best Wish
> > > > > > > > > > > — Kerwin (zhuangchong)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I am here to propose Apache Dolphinscheduler version
> > 3.2.0
> > > > plan to
> > > > > > > > > > > > release in July 2023 or August 2023.
> > > > > > > > > > > >
> > > > > > > > > > > > Actually, I already created a new branch named
> > > > `3.2.0-prepare` for
> > > > > > > > > > > > it[1], including all features merged to branch dev
> > after
> > > > we release
> > > > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting
> > > > functional
> > > > > > > > > > > > additions and code base refactoring.
> > > > > > > > > > > >
> > > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now,
> > anyone
> > > > interested
> > > > > > > > > > > > to Dolphinscheduler or want to contribute to the latest
> > > > version is
> > > > > > > > > > > > welcome to test it. You can checkout the
> > 3.2.0-prepare[1]
> > > > branch and
> > > > > > > > > > > > start the standalone server, or build tarball for
> > cluster
> > > > mode test
> > > > > > > > > > > > based on your wish. In this version, we need to test
> > the
> > > > core function
> > > > > > > > > > > > of dolphinscheduler like CURD of project, workflow,
> > task
> > > > and execute
> > > > > > > > > > > > them. we allow need to test the resource center
> > carefully
> > > > because it
> > > > > > > > > > > > changes a lot in this version.
> > > > > > > > > > > >
> > > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any
> > > > bugs. We have
> > > > > > > > > > > > a specific version for this incoming version name
> > > > `3.2.0-prepare` in
> > > > > > > > > > > > the bugs reported template.
> > > > > > > > > > > >
> > > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is
> > > > reported by others or
> > > > > > > > > > > > you
> > > > > > > > > > > >
> > > > > > > > > > > > Some of the functional features list for 3.2.0:
> > > > > > > > > > > >
> > > > > > > > > > > > * User friendly
> > > > > > > > > > > >     * UI only: More hyperlinks jump between resource,
> > > > adjustable column
> > > > > > > > > > > > width
> > > > > > > > > > > >     * Default tenant: Out of box run workflow by
> > default
> > > > tenant
> > > > > > > > > > > >     * Run or rerun task in task instance
> > > > > > > > > > > >     * Human readable json export result
> > > > > > > > > > > > * Cloud native
> > > > > > > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > > > > > > >     * Support KEDA autoscaler worker
> > > > > > > > > > > >     * Support Terraform deploy to AWS
> > > > > > > > > > > >     * Multi arch for zookeper and postgresql(all
> > services
> > > > are supported
> > > > > > > > > > > > now)
> > > > > > > > > > > > * Resource center
> > > > > > > > > > > >     * Refactor resource center, we use provider API to
> > > > list and manage
> > > > > > > > > > > > resource now
> > > > > > > > > > > >     * Allow resource re-upload now
> > > > > > > > > > > > * Datasource
> > > > > > > > > > > >     * New datasource:
> > > > > > > > > > > >
> > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > > > > > > Athena
> > > > > > > > > > > >     * Support access key for redshift datasource
> > > > > > > > > > > > * Tasks
> > > > > > > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database
> > > > Migration、Java Task
> > > > > > > > > > > >     * Improve for exists: Add presto for datax,
> > parameter
> > > > passing for
> > > > > > > > > > > > python and http, sql task multiple sql delimiter,
> > > > dependent depend
> > > > > > > > > > > > itself
> > > > > > > > > > > >     * Introduce task cache for MLops or other long-run
> > > > tasks
> > > > > > > > > > > > * Parameter
> > > > > > > > > > > >     * Add new rule of build-in function
> > > > > > > > > > > >     * Introduce file type parameter
> > > > > > > > > > > >     * Startup parameters have the highest priority
> > > > > > > > > > > > * Logs: Introduce remote logs mechanism and support
> > Google
> > > > Cloud
> > > > > > > > > > > > Storage、S3、OSS up to now
> > > > > > > > > > > > * API:
> > > > > > > > > > > >     * Introduce some new restful API(experimental),
> > > > including
> > > > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow
> > and
> > > > > > > > > > > > schedule、task relation
> > > > > > > > > > > >     * Can get task instance id when triggering workflow
> > > > via API
> > > > > > > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > > > > > > [2]:
> > > > https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Best Wish
> > > > > > > > > > > > —Jay Chung
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best Wish
> > > > > > > > > > —Jay Chung
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Wish
> > > > > > > > > —Jay Chung
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Wish
> > > > > > > > —Jay Chung
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > —Jay Chung
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > > >
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
> >



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by eye gu <gu...@gmail.com>.

Just merge the dependent third-party packages into the same directory, and
each service directory is still separate. Just modify the directory
specified in the startup script for dependencies

Jay Chung <zh...@gmail.com> 于2023年9月11日周一 09:49写道：

> Not yet, but we should not merge them into one single directory, cause
> we take time to separate our services into multiple directories for
> easy access and use
>
> On Sun, Sep 10, 2023 at 12:15 PM 顾泽民 <gu...@gmail.com> wrote:
> >
> > I have an idea, can we merge libs from a single directory into one
> > directory?
> >
> > Jay Chung <zh...@gmail.com> 于2023年9月7日周四 17:14写道：
> >
> > > Hi guys,
> > >
> > > !!! We need some hero for the tarball size of dolphinscheduler. !!!
> > > ASF infra team disagrees with the Jira ticket for bumping our SVN
> > > limit[1], we need to find some way to make our binary tarball smaller
> > > than 950MB. and the current size of our tarball is almost 1.2GB which
> > > means we should reduce at least 278MB for the tarball.
> > >
> > > We have a DSIP to separate and release plugins in the future[2], but
> > > it is still WIP, maybe we have to find another way to reduce the size
> > > of binary tarball
> > >
> > > Please let me know if you have any good ideas about our tarball size,
> > > thanks a lot
> > >
> > > [1]: https://issues.apache.org/jira/browse/INFRA-24944
> > > [2]: https://github.com/apache/dolphinscheduler/issues/1252
> > >
> > > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zh...@gmail.com>
> > > wrote:
> > > >
> > > > And if you find some bugfix still WIP and nearly to merge, please let
> > > > me know in this thread, maybe we could add them to 3.2.0 too, cause
> we
> > > > block by SVN limit now
> > > >
> > > > Some of our community found out about two of them, and thank
> Gallardot
> > > > for bringing those up
> > > >
> > > > https://github.com/apache/dolphinscheduler/pull/14328
> > > > https://github.com/apache/dolphinscheduler/pull/14829
> > > >
> > > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com>
> > > wrote:
> > > > >
> > > > > FYI
> > > > >
> > > > > I use `gh` cli to batch bump all issues and prs with milestone
> `3.2.0`
> > > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> > > > > (especially bug). so please take care of the bug with milestone
> > > > > `3.3.0` and still open since Aug 31, 2023
> > > > >
> > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <
> zhongjiajie955@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Version 3.2.0 is nearly close and released during those days,
> but we
> > > > > > can not upload it to Apache SVN because too big for our tarball,
> We
> > > > > > created a Jira ticket to bump the svn limit for dolphinscheduler
> and
> > > > > > it is still in discussion. I will follow up when it has a further
> > > > > > update
> > > > > >
> > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <
> zhongjiajie955@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I had cherry-picked some recent commits to the branch
> > > 3.2.0-prepare,
> > > > > > > anyone who interested in checking 3.2.0 features could clone
> the
> > > > > > > latest branch to have a look
> > > > > > >
> > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <
> > > zhongjiajie955@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > I submit new PR[1] to remove test datasource in version
> 3.2.0,
> > > in [2]
> > > > > > > > we introduce the test task and test datasource. It is a good
> > > feature
> > > > > > > > to allow user run in test environment.
> > > > > > > >
> > > > > > > > But I find out we have to create test datasource before
> create
> > > the
> > > > > > > > prod one. I think it make user confuse especial some one who
> do
> > > not
> > > > > > > > use test environment. So I think we should remove it in
> version
> > > 3.2.0.
> > > > > > > >
> > > > > > > > If you have different idea, please let me know in this mail
> > > thread
> > > > > > > >
> > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > > > > > >
> > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <
> > > zhongjiajie955@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Community,
> > > > > > > > >
> > > > > > > > > I added a checklist for the version 3.2.0 release,
> everyone who
> > > > > > > > > interested in the incoming version can participate in our
> > > test, the
> > > > > > > > > detail checklist issue is in
> > > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > > > > > >
> > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <
> > > zhuang.kerwin@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks Jay.
> > > > > > > > > >
> > > > > > > > > > Looking forward to the 3.2.0 release.
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best Wish
> > > > > > > > > > — Kerwin (zhuangchong)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I am here to propose Apache Dolphinscheduler version
> 3.2.0
> > > plan to
> > > > > > > > > > > release in July 2023 or August 2023.
> > > > > > > > > > >
> > > > > > > > > > > Actually, I already created a new branch named
> > > `3.2.0-prepare` for
> > > > > > > > > > > it[1], including all features merged to branch dev
> after
> > > we release
> > > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting
> > > functional
> > > > > > > > > > > additions and code base refactoring.
> > > > > > > > > > >
> > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now,
> anyone
> > > interested
> > > > > > > > > > > to Dolphinscheduler or want to contribute to the latest
> > > version is
> > > > > > > > > > > welcome to test it. You can checkout the
> 3.2.0-prepare[1]
> > > branch and
> > > > > > > > > > > start the standalone server, or build tarball for
> cluster
> > > mode test
> > > > > > > > > > > based on your wish. In this version, we need to test
> the
> > > core function
> > > > > > > > > > > of dolphinscheduler like CURD of project, workflow,
> task
> > > and execute
> > > > > > > > > > > them. we allow need to test the resource center
> carefully
> > > because it
> > > > > > > > > > > changes a lot in this version.
> > > > > > > > > > >
> > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any
> > > bugs. We have
> > > > > > > > > > > a specific version for this incoming version name
> > > `3.2.0-prepare` in
> > > > > > > > > > > the bugs reported template.
> > > > > > > > > > >
> > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is
> > > reported by others or
> > > > > > > > > > > you
> > > > > > > > > > >
> > > > > > > > > > > Some of the functional features list for 3.2.0:
> > > > > > > > > > >
> > > > > > > > > > > * User friendly
> > > > > > > > > > >     * UI only: More hyperlinks jump between resource,
> > > adjustable column
> > > > > > > > > > > width
> > > > > > > > > > >     * Default tenant: Out of box run workflow by
> default
> > > tenant
> > > > > > > > > > >     * Run or rerun task in task instance
> > > > > > > > > > >     * Human readable json export result
> > > > > > > > > > > * Cloud native
> > > > > > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > > > > > >     * Support KEDA autoscaler worker
> > > > > > > > > > >     * Support Terraform deploy to AWS
> > > > > > > > > > >     * Multi arch for zookeper and postgresql(all
> services
> > > are supported
> > > > > > > > > > > now)
> > > > > > > > > > > * Resource center
> > > > > > > > > > >     * Refactor resource center, we use provider API to
> > > list and manage
> > > > > > > > > > > resource now
> > > > > > > > > > >     * Allow resource re-upload now
> > > > > > > > > > > * Datasource
> > > > > > > > > > >     * New datasource:
> > > > > > > > > > >
> > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > > > > > Athena
> > > > > > > > > > >     * Support access key for redshift datasource
> > > > > > > > > > > * Tasks
> > > > > > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database
> > > Migration、Java Task
> > > > > > > > > > >     * Improve for exists: Add presto for datax,
> parameter
> > > passing for
> > > > > > > > > > > python and http, sql task multiple sql delimiter,
> > > dependent depend
> > > > > > > > > > > itself
> > > > > > > > > > >     * Introduce task cache for MLops or other long-run
> > > tasks
> > > > > > > > > > > * Parameter
> > > > > > > > > > >     * Add new rule of build-in function
> > > > > > > > > > >     * Introduce file type parameter
> > > > > > > > > > >     * Startup parameters have the highest priority
> > > > > > > > > > > * Logs: Introduce remote logs mechanism and support
> Google
> > > Cloud
> > > > > > > > > > > Storage、S3、OSS up to now
> > > > > > > > > > > * API:
> > > > > > > > > > >     * Introduce some new restful API(experimental),
> > > including
> > > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow
> and
> > > > > > > > > > > schedule、task relation
> > > > > > > > > > >     * Can get task instance id when triggering workflow
> > > via API
> > > > > > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > > > > > [2]:
> > > https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best Wish
> > > > > > > > > > > —Jay Chung
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Wish
> > > > > > > > > —Jay Chung
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Wish
> > > > > > > > —Jay Chung
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > —Jay Chung
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> > >
>
>
>
> --
> Best Wish
> —Jay Chung
>

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Not yet, but we should not merge them into one single directory, cause
we take time to separate our services into multiple directories for
easy access and use

On Sun, Sep 10, 2023 at 12:15 PM 顾泽民 <gu...@gmail.com> wrote:
>
> I have an idea, can we merge libs from a single directory into one
> directory?
>
> Jay Chung <zh...@gmail.com> 于2023年9月7日周四 17:14写道：
>
> > Hi guys,
> >
> > !!! We need some hero for the tarball size of dolphinscheduler. !!!
> > ASF infra team disagrees with the Jira ticket for bumping our SVN
> > limit[1], we need to find some way to make our binary tarball smaller
> > than 950MB. and the current size of our tarball is almost 1.2GB which
> > means we should reduce at least 278MB for the tarball.
> >
> > We have a DSIP to separate and release plugins in the future[2], but
> > it is still WIP, maybe we have to find another way to reduce the size
> > of binary tarball
> >
> > Please let me know if you have any good ideas about our tarball size,
> > thanks a lot
> >
> > [1]: https://issues.apache.org/jira/browse/INFRA-24944
> > [2]: https://github.com/apache/dolphinscheduler/issues/1252
> >
> > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zh...@gmail.com>
> > wrote:
> > >
> > > And if you find some bugfix still WIP and nearly to merge, please let
> > > me know in this thread, maybe we could add them to 3.2.0 too, cause we
> > > block by SVN limit now
> > >
> > > Some of our community found out about two of them, and thank Gallardot
> > > for bringing those up
> > >
> > > https://github.com/apache/dolphinscheduler/pull/14328
> > > https://github.com/apache/dolphinscheduler/pull/14829
> > >
> > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com>
> > wrote:
> > > >
> > > > FYI
> > > >
> > > > I use `gh` cli to batch bump all issues and prs with milestone `3.2.0`
> > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> > > > (especially bug). so please take care of the bug with milestone
> > > > `3.3.0` and still open since Aug 31, 2023
> > > >
> > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zh...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Version 3.2.0 is nearly close and released during those days, but we
> > > > > can not upload it to Apache SVN because too big for our tarball, We
> > > > > created a Jira ticket to bump the svn limit for dolphinscheduler and
> > > > > it is still in discussion. I will follow up when it has a further
> > > > > update
> > > > >
> > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I had cherry-picked some recent commits to the branch
> > 3.2.0-prepare,
> > > > > > anyone who interested in checking 3.2.0 features could clone the
> > > > > > latest branch to have a look
> > > > > >
> > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <
> > zhongjiajie955@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > I submit new PR[1] to remove test datasource in version 3.2.0,
> > in [2]
> > > > > > > we introduce the test task and test datasource. It is a good
> > feature
> > > > > > > to allow user run in test environment.
> > > > > > >
> > > > > > > But I find out we have to create test datasource before create
> > the
> > > > > > > prod one. I think it make user confuse especial some one who do
> > not
> > > > > > > use test environment. So I think we should remove it in version
> > 3.2.0.
> > > > > > >
> > > > > > > If you have different idea, please let me know in this mail
> > thread
> > > > > > >
> > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > > > > >
> > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <
> > zhongjiajie955@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Community,
> > > > > > > >
> > > > > > > > I added a checklist for the version 3.2.0 release, everyone who
> > > > > > > > interested in the incoming version can participate in our
> > test, the
> > > > > > > > detail checklist issue is in
> > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > > > > >
> > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <
> > zhuang.kerwin@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks Jay.
> > > > > > > > >
> > > > > > > > > Looking forward to the 3.2.0 release.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Wish
> > > > > > > > > — Kerwin (zhuangchong)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0
> > plan to
> > > > > > > > > > release in July 2023 or August 2023.
> > > > > > > > > >
> > > > > > > > > > Actually, I already created a new branch named
> > `3.2.0-prepare` for
> > > > > > > > > > it[1], including all features merged to branch dev after
> > we release
> > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting
> > functional
> > > > > > > > > > additions and code base refactoring.
> > > > > > > > > >
> > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone
> > interested
> > > > > > > > > > to Dolphinscheduler or want to contribute to the latest
> > version is
> > > > > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1]
> > branch and
> > > > > > > > > > start the standalone server, or build tarball for cluster
> > mode test
> > > > > > > > > > based on your wish. In this version, we need to test the
> > core function
> > > > > > > > > > of dolphinscheduler like CURD of project, workflow, task
> > and execute
> > > > > > > > > > them. we allow need to test the resource center carefully
> > because it
> > > > > > > > > > changes a lot in this version.
> > > > > > > > > >
> > > > > > > > > > You can report dolpinscheduler issue[2] if you find any
> > bugs. We have
> > > > > > > > > > a specific version for this incoming version name
> > `3.2.0-prepare` in
> > > > > > > > > > the bugs reported template.
> > > > > > > > > >
> > > > > > > > > > And feel free to submit PR to fix bugs, whatever is
> > reported by others or
> > > > > > > > > > you
> > > > > > > > > >
> > > > > > > > > > Some of the functional features list for 3.2.0:
> > > > > > > > > >
> > > > > > > > > > * User friendly
> > > > > > > > > >     * UI only: More hyperlinks jump between resource,
> > adjustable column
> > > > > > > > > > width
> > > > > > > > > >     * Default tenant: Out of box run workflow by default
> > tenant
> > > > > > > > > >     * Run or rerun task in task instance
> > > > > > > > > >     * Human readable json export result
> > > > > > > > > > * Cloud native
> > > > > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > > > > >     * Support KEDA autoscaler worker
> > > > > > > > > >     * Support Terraform deploy to AWS
> > > > > > > > > >     * Multi arch for zookeper and postgresql(all services
> > are supported
> > > > > > > > > > now)
> > > > > > > > > > * Resource center
> > > > > > > > > >     * Refactor resource center, we use provider API to
> > list and manage
> > > > > > > > > > resource now
> > > > > > > > > >     * Allow resource re-upload now
> > > > > > > > > > * Datasource
> > > > > > > > > >     * New datasource:
> > > > > > > > > >
> > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > > > > Athena
> > > > > > > > > >     * Support access key for redshift datasource
> > > > > > > > > > * Tasks
> > > > > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database
> > Migration、Java Task
> > > > > > > > > >     * Improve for exists: Add presto for datax, parameter
> > passing for
> > > > > > > > > > python and http, sql task multiple sql delimiter,
> > dependent depend
> > > > > > > > > > itself
> > > > > > > > > >     * Introduce task cache for MLops or other long-run
> > tasks
> > > > > > > > > > * Parameter
> > > > > > > > > >     * Add new rule of build-in function
> > > > > > > > > >     * Introduce file type parameter
> > > > > > > > > >     * Startup parameters have the highest priority
> > > > > > > > > > * Logs: Introduce remote logs mechanism and support Google
> > Cloud
> > > > > > > > > > Storage、S3、OSS up to now
> > > > > > > > > > * API:
> > > > > > > > > >     * Introduce some new restful API(experimental),
> > including
> > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > > > > > > schedule、task relation
> > > > > > > > > >     * Can get task instance id when triggering workflow
> > via API
> > > > > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > > > > >
> > > > > > > > > > [1]:
> > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > > > > [2]:
> > https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best Wish
> > > > > > > > > > —Jay Chung
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Wish
> > > > > > > > —Jay Chung
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > —Jay Chung
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
> >



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by 顾泽民 <gu...@gmail.com>.

I have an idea, can we merge libs from a single directory into one
directory?

Jay Chung <zh...@gmail.com> 于2023年9月7日周四 17:14写道：

> Hi guys,
>
> !!! We need some hero for the tarball size of dolphinscheduler. !!!
> ASF infra team disagrees with the Jira ticket for bumping our SVN
> limit[1], we need to find some way to make our binary tarball smaller
> than 950MB. and the current size of our tarball is almost 1.2GB which
> means we should reduce at least 278MB for the tarball.
>
> We have a DSIP to separate and release plugins in the future[2], but
> it is still WIP, maybe we have to find another way to reduce the size
> of binary tarball
>
> Please let me know if you have any good ideas about our tarball size,
> thanks a lot
>
> [1]: https://issues.apache.org/jira/browse/INFRA-24944
> [2]: https://github.com/apache/dolphinscheduler/issues/1252
>
> On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zh...@gmail.com>
> wrote:
> >
> > And if you find some bugfix still WIP and nearly to merge, please let
> > me know in this thread, maybe we could add them to 3.2.0 too, cause we
> > block by SVN limit now
> >
> > Some of our community found out about two of them, and thank Gallardot
> > for bringing those up
> >
> > https://github.com/apache/dolphinscheduler/pull/14328
> > https://github.com/apache/dolphinscheduler/pull/14829
> >
> > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com>
> wrote:
> > >
> > > FYI
> > >
> > > I use `gh` cli to batch bump all issues and prs with milestone `3.2.0`
> > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> > > (especially bug). so please take care of the bug with milestone
> > > `3.3.0` and still open since Aug 31, 2023
> > >
> > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zh...@gmail.com>
> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Version 3.2.0 is nearly close and released during those days, but we
> > > > can not upload it to Apache SVN because too big for our tarball, We
> > > > created a Jira ticket to bump the svn limit for dolphinscheduler and
> > > > it is still in discussion. I will follow up when it has a further
> > > > update
> > > >
> > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I had cherry-picked some recent commits to the branch
> 3.2.0-prepare,
> > > > > anyone who interested in checking 3.2.0 features could clone the
> > > > > latest branch to have a look
> > > > >
> > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <
> zhongjiajie955@gmail.com> wrote:
> > > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I submit new PR[1] to remove test datasource in version 3.2.0,
> in [2]
> > > > > > we introduce the test task and test datasource. It is a good
> feature
> > > > > > to allow user run in test environment.
> > > > > >
> > > > > > But I find out we have to create test datasource before create
> the
> > > > > > prod one. I think it make user confuse especial some one who do
> not
> > > > > > use test environment. So I think we should remove it in version
> 3.2.0.
> > > > > >
> > > > > > If you have different idea, please let me know in this mail
> thread
> > > > > >
> > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > > > >
> > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <
> zhongjiajie955@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Community,
> > > > > > >
> > > > > > > I added a checklist for the version 3.2.0 release, everyone who
> > > > > > > interested in the incoming version can participate in our
> test, the
> > > > > > > detail checklist issue is in
> > > > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > > > >
> > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <
> zhuang.kerwin@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Thanks Jay.
> > > > > > > >
> > > > > > > > Looking forward to the 3.2.0 release.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Wish
> > > > > > > > — Kerwin (zhuangchong)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0
> plan to
> > > > > > > > > release in July 2023 or August 2023.
> > > > > > > > >
> > > > > > > > > Actually, I already created a new branch named
> `3.2.0-prepare` for
> > > > > > > > > it[1], including all features merged to branch dev after
> we release
> > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting
> functional
> > > > > > > > > additions and code base refactoring.
> > > > > > > > >
> > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone
> interested
> > > > > > > > > to Dolphinscheduler or want to contribute to the latest
> version is
> > > > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1]
> branch and
> > > > > > > > > start the standalone server, or build tarball for cluster
> mode test
> > > > > > > > > based on your wish. In this version, we need to test the
> core function
> > > > > > > > > of dolphinscheduler like CURD of project, workflow, task
> and execute
> > > > > > > > > them. we allow need to test the resource center carefully
> because it
> > > > > > > > > changes a lot in this version.
> > > > > > > > >
> > > > > > > > > You can report dolpinscheduler issue[2] if you find any
> bugs. We have
> > > > > > > > > a specific version for this incoming version name
> `3.2.0-prepare` in
> > > > > > > > > the bugs reported template.
> > > > > > > > >
> > > > > > > > > And feel free to submit PR to fix bugs, whatever is
> reported by others or
> > > > > > > > > you
> > > > > > > > >
> > > > > > > > > Some of the functional features list for 3.2.0:
> > > > > > > > >
> > > > > > > > > * User friendly
> > > > > > > > >     * UI only: More hyperlinks jump between resource,
> adjustable column
> > > > > > > > > width
> > > > > > > > >     * Default tenant: Out of box run workflow by default
> tenant
> > > > > > > > >     * Run or rerun task in task instance
> > > > > > > > >     * Human readable json export result
> > > > > > > > > * Cloud native
> > > > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > > > >     * Support KEDA autoscaler worker
> > > > > > > > >     * Support Terraform deploy to AWS
> > > > > > > > >     * Multi arch for zookeper and postgresql(all services
> are supported
> > > > > > > > > now)
> > > > > > > > > * Resource center
> > > > > > > > >     * Refactor resource center, we use provider API to
> list and manage
> > > > > > > > > resource now
> > > > > > > > >     * Allow resource re-upload now
> > > > > > > > > * Datasource
> > > > > > > > >     * New datasource:
> > > > > > > > >
> snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > > > Athena
> > > > > > > > >     * Support access key for redshift datasource
> > > > > > > > > * Tasks
> > > > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database
> Migration、Java Task
> > > > > > > > >     * Improve for exists: Add presto for datax, parameter
> passing for
> > > > > > > > > python and http, sql task multiple sql delimiter,
> dependent depend
> > > > > > > > > itself
> > > > > > > > >     * Introduce task cache for MLops or other long-run
> tasks
> > > > > > > > > * Parameter
> > > > > > > > >     * Add new rule of build-in function
> > > > > > > > >     * Introduce file type parameter
> > > > > > > > >     * Startup parameters have the highest priority
> > > > > > > > > * Logs: Introduce remote logs mechanism and support Google
> Cloud
> > > > > > > > > Storage、S3、OSS up to now
> > > > > > > > > * API:
> > > > > > > > >     * Introduce some new restful API(experimental),
> including
> > > > > > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > > > > > schedule、task relation
> > > > > > > > >     * Can get task instance id when triggering workflow
> via API
> > > > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > > > >
> > > > > > > > > [1]:
> https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > > > [2]:
> https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Wish
> > > > > > > > > —Jay Chung
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > —Jay Chung
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung
>

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Hi guys,

!!! We need some hero for the tarball size of dolphinscheduler. !!!
ASF infra team disagrees with the Jira ticket for bumping our SVN
limit[1], we need to find some way to make our binary tarball smaller
than 950MB. and the current size of our tarball is almost 1.2GB which
means we should reduce at least 278MB for the tarball.

We have a DSIP to separate and release plugins in the future[2], but
it is still WIP, maybe we have to find another way to reduce the size
of binary tarball

Please let me know if you have any good ideas about our tarball size,
thanks a lot

[1]: https://issues.apache.org/jira/browse/INFRA-24944
[2]: https://github.com/apache/dolphinscheduler/issues/1252

On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zh...@gmail.com> wrote:
>
> And if you find some bugfix still WIP and nearly to merge, please let
> me know in this thread, maybe we could add them to 3.2.0 too, cause we
> block by SVN limit now
>
> Some of our community found out about two of them, and thank Gallardot
> for bringing those up
>
> https://github.com/apache/dolphinscheduler/pull/14328
> https://github.com/apache/dolphinscheduler/pull/14829
>
> On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com> wrote:
> >
> > FYI
> >
> > I use `gh` cli to batch bump all issues and prs with milestone `3.2.0`
> > but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> > (especially bug). so please take care of the bug with milestone
> > `3.3.0` and still open since Aug 31, 2023
> >
> > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zh...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > Version 3.2.0 is nearly close and released during those days, but we
> > > can not upload it to Apache SVN because too big for our tarball, We
> > > created a Jira ticket to bump the svn limit for dolphinscheduler and
> > > it is still in discussion. I will follow up when it has a further
> > > update
> > >
> > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I had cherry-picked some recent commits to the branch 3.2.0-prepare,
> > > > anyone who interested in checking 3.2.0 features could clone the
> > > > latest branch to have a look
> > > >
> > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zh...@gmail.com> wrote:
> > > > >
> > > > > Hi guys,
> > > > >
> > > > > I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
> > > > > we introduce the test task and test datasource. It is a good feature
> > > > > to allow user run in test environment.
> > > > >
> > > > > But I find out we have to create test datasource before create the
> > > > > prod one. I think it make user confuse especial some one who do not
> > > > > use test environment. So I think we should remove it in version 3.2.0.
> > > > >
> > > > > If you have different idea, please let me know in this mail thread
> > > > >
> > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > > >
> > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
> > > > > >
> > > > > > Hi Community,
> > > > > >
> > > > > > I added a checklist for the version 3.2.0 release, everyone who
> > > > > > interested in the incoming version can participate in our test, the
> > > > > > detail checklist issue is in
> > > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > > >
> > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks Jay.
> > > > > > >
> > > > > > > Looking forward to the 3.2.0 release.
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > — Kerwin (zhuangchong)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > > > > > > release in July 2023 or August 2023.
> > > > > > > >
> > > > > > > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > > > > > > it[1], including all features merged to branch dev after we release
> > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > > > > > > additions and code base refactoring.
> > > > > > > >
> > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > > > > > > to Dolphinscheduler or want to contribute to the latest version is
> > > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > > > > > > start the standalone server, or build tarball for cluster mode test
> > > > > > > > based on your wish. In this version, we need to test the core function
> > > > > > > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > > > > > > them. we allow need to test the resource center carefully because it
> > > > > > > > changes a lot in this version.
> > > > > > > >
> > > > > > > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > > > > > > a specific version for this incoming version name `3.2.0-prepare` in
> > > > > > > > the bugs reported template.
> > > > > > > >
> > > > > > > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > > > > > > you
> > > > > > > >
> > > > > > > > Some of the functional features list for 3.2.0:
> > > > > > > >
> > > > > > > > * User friendly
> > > > > > > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > > > > > > width
> > > > > > > >     * Default tenant: Out of box run workflow by default tenant
> > > > > > > >     * Run or rerun task in task instance
> > > > > > > >     * Human readable json export result
> > > > > > > > * Cloud native
> > > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > > >     * Support KEDA autoscaler worker
> > > > > > > >     * Support Terraform deploy to AWS
> > > > > > > >     * Multi arch for zookeper and postgresql(all services are supported
> > > > > > > > now)
> > > > > > > > * Resource center
> > > > > > > >     * Refactor resource center, we use provider API to list and manage
> > > > > > > > resource now
> > > > > > > >     * Allow resource re-upload now
> > > > > > > > * Datasource
> > > > > > > >     * New datasource:
> > > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > > Athena
> > > > > > > >     * Support access key for redshift datasource
> > > > > > > > * Tasks
> > > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > > > > > > >     * Improve for exists: Add presto for datax, parameter passing for
> > > > > > > > python and http, sql task multiple sql delimiter, dependent depend
> > > > > > > > itself
> > > > > > > >     * Introduce task cache for MLops or other long-run tasks
> > > > > > > > * Parameter
> > > > > > > >     * Add new rule of build-in function
> > > > > > > >     * Introduce file type parameter
> > > > > > > >     * Startup parameters have the highest priority
> > > > > > > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > > > > > > Storage、S3、OSS up to now
> > > > > > > > * API:
> > > > > > > >     * Introduce some new restful API(experimental), including
> > > > > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > > > > schedule、task relation
> > > > > > > >     * Can get task instance id when triggering workflow via API
> > > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > > >
> > > > > > > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Wish
> > > > > > > > —Jay Chung
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

And if you find some bugfix still WIP and nearly to merge, please let
me know in this thread, maybe we could add them to 3.2.0 too, cause we
block by SVN limit now

Some of our community found out about two of them, and thank Gallardot
for bringing those up

https://github.com/apache/dolphinscheduler/pull/14328
https://github.com/apache/dolphinscheduler/pull/14829

On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zh...@gmail.com> wrote:
>
> FYI
>
> I use `gh` cli to batch bump all issues and prs with milestone `3.2.0`
> but still open to `3.3.0`, maybe some of then should tag in 3.2.1
> (especially bug). so please take care of the bug with milestone
> `3.3.0` and still open since Aug 31, 2023
>
> On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zh...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Version 3.2.0 is nearly close and released during those days, but we
> > can not upload it to Apache SVN because too big for our tarball, We
> > created a Jira ticket to bump the svn limit for dolphinscheduler and
> > it is still in discussion. I will follow up when it has a further
> > update
> >
> > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > I had cherry-picked some recent commits to the branch 3.2.0-prepare,
> > > anyone who interested in checking 3.2.0 features could clone the
> > > latest branch to have a look
> > >
> > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zh...@gmail.com> wrote:
> > > >
> > > > Hi guys,
> > > >
> > > > I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
> > > > we introduce the test task and test datasource. It is a good feature
> > > > to allow user run in test environment.
> > > >
> > > > But I find out we have to create test datasource before create the
> > > > prod one. I think it make user confuse especial some one who do not
> > > > use test environment. So I think we should remove it in version 3.2.0.
> > > >
> > > > If you have different idea, please let me know in this mail thread
> > > >
> > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > > >
> > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
> > > > >
> > > > > Hi Community,
> > > > >
> > > > > I added a checklist for the version 3.2.0 release, everyone who
> > > > > interested in the incoming version can participate in our test, the
> > > > > detail checklist issue is in
> > > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > > >
> > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Jay.
> > > > > >
> > > > > > Looking forward to the 3.2.0 release.
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > — Kerwin (zhuangchong)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > > > > > release in July 2023 or August 2023.
> > > > > > >
> > > > > > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > > > > > it[1], including all features merged to branch dev after we release
> > > > > > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > > > > > additions and code base refactoring.
> > > > > > >
> > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > > > > > to Dolphinscheduler or want to contribute to the latest version is
> > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > > > > > start the standalone server, or build tarball for cluster mode test
> > > > > > > based on your wish. In this version, we need to test the core function
> > > > > > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > > > > > them. we allow need to test the resource center carefully because it
> > > > > > > changes a lot in this version.
> > > > > > >
> > > > > > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > > > > > a specific version for this incoming version name `3.2.0-prepare` in
> > > > > > > the bugs reported template.
> > > > > > >
> > > > > > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > > > > > you
> > > > > > >
> > > > > > > Some of the functional features list for 3.2.0:
> > > > > > >
> > > > > > > * User friendly
> > > > > > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > > > > > width
> > > > > > >     * Default tenant: Out of box run workflow by default tenant
> > > > > > >     * Run or rerun task in task instance
> > > > > > >     * Human readable json export result
> > > > > > > * Cloud native
> > > > > > >     * Kubernetes Real-time log and custom label support
> > > > > > >     * Support KEDA autoscaler worker
> > > > > > >     * Support Terraform deploy to AWS
> > > > > > >     * Multi arch for zookeper and postgresql(all services are supported
> > > > > > > now)
> > > > > > > * Resource center
> > > > > > >     * Refactor resource center, we use provider API to list and manage
> > > > > > > resource now
> > > > > > >     * Allow resource re-upload now
> > > > > > > * Datasource
> > > > > > >     * New datasource:
> > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > > Athena
> > > > > > >     * Support access key for redshift datasource
> > > > > > > * Tasks
> > > > > > >     * New tasks: azure datafactory、apache
> > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > > > > > >     * Improve for exists: Add presto for datax, parameter passing for
> > > > > > > python and http, sql task multiple sql delimiter, dependent depend
> > > > > > > itself
> > > > > > >     * Introduce task cache for MLops or other long-run tasks
> > > > > > > * Parameter
> > > > > > >     * Add new rule of build-in function
> > > > > > >     * Introduce file type parameter
> > > > > > >     * Startup parameters have the highest priority
> > > > > > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > > > > > Storage、S3、OSS up to now
> > > > > > > * API:
> > > > > > >     * Introduce some new restful API(experimental), including
> > > > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > > > schedule、task relation
> > > > > > >     * Can get task instance id when triggering workflow via API
> > > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > > >
> > > > > > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > > >
> > > > > > > --
> > > > > > > Best Wish
> > > > > > > —Jay Chung
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

FYI

I use `gh` cli to batch bump all issues and prs with milestone `3.2.0`
but still open to `3.3.0`, maybe some of then should tag in 3.2.1
(especially bug). so please take care of the bug with milestone
`3.3.0` and still open since Aug 31, 2023

On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zh...@gmail.com> wrote:
>
> Hi all,
>
> Version 3.2.0 is nearly close and released during those days, but we
> can not upload it to Apache SVN because too big for our tarball, We
> created a Jira ticket to bump the svn limit for dolphinscheduler and
> it is still in discussion. I will follow up when it has a further
> update
>
> On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I had cherry-picked some recent commits to the branch 3.2.0-prepare,
> > anyone who interested in checking 3.2.0 features could clone the
> > latest branch to have a look
> >
> > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zh...@gmail.com> wrote:
> > >
> > > Hi guys,
> > >
> > > I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
> > > we introduce the test task and test datasource. It is a good feature
> > > to allow user run in test environment.
> > >
> > > But I find out we have to create test datasource before create the
> > > prod one. I think it make user confuse especial some one who do not
> > > use test environment. So I think we should remove it in version 3.2.0.
> > >
> > > If you have different idea, please let me know in this mail thread
> > >
> > > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> > >
> > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
> > > >
> > > > Hi Community,
> > > >
> > > > I added a checklist for the version 3.2.0 release, everyone who
> > > > interested in the incoming version can participate in our test, the
> > > > detail checklist issue is in
> > > > https://github.com/apache/dolphinscheduler/issues/14372
> > > >
> > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> > > > >
> > > > > Thanks Jay.
> > > > >
> > > > > Looking forward to the 3.2.0 release.
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > — Kerwin (zhuangchong)
> > > > >
> > > > >
> > > > >
> > > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > > > > release in July 2023 or August 2023.
> > > > > >
> > > > > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > > > > it[1], including all features merged to branch dev after we release
> > > > > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > > > > additions and code base refactoring.
> > > > > >
> > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > > > > to Dolphinscheduler or want to contribute to the latest version is
> > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > > > > start the standalone server, or build tarball for cluster mode test
> > > > > > based on your wish. In this version, we need to test the core function
> > > > > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > > > > them. we allow need to test the resource center carefully because it
> > > > > > changes a lot in this version.
> > > > > >
> > > > > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > > > > a specific version for this incoming version name `3.2.0-prepare` in
> > > > > > the bugs reported template.
> > > > > >
> > > > > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > > > > you
> > > > > >
> > > > > > Some of the functional features list for 3.2.0:
> > > > > >
> > > > > > * User friendly
> > > > > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > > > > width
> > > > > >     * Default tenant: Out of box run workflow by default tenant
> > > > > >     * Run or rerun task in task instance
> > > > > >     * Human readable json export result
> > > > > > * Cloud native
> > > > > >     * Kubernetes Real-time log and custom label support
> > > > > >     * Support KEDA autoscaler worker
> > > > > >     * Support Terraform deploy to AWS
> > > > > >     * Multi arch for zookeper and postgresql(all services are supported
> > > > > > now)
> > > > > > * Resource center
> > > > > >     * Refactor resource center, we use provider API to list and manage
> > > > > > resource now
> > > > > >     * Allow resource re-upload now
> > > > > > * Datasource
> > > > > >     * New datasource:
> > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > > Athena
> > > > > >     * Support access key for redshift datasource
> > > > > > * Tasks
> > > > > >     * New tasks: azure datafactory、apache
> > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > > > > >     * Improve for exists: Add presto for datax, parameter passing for
> > > > > > python and http, sql task multiple sql delimiter, dependent depend
> > > > > > itself
> > > > > >     * Introduce task cache for MLops or other long-run tasks
> > > > > > * Parameter
> > > > > >     * Add new rule of build-in function
> > > > > >     * Introduce file type parameter
> > > > > >     * Startup parameters have the highest priority
> > > > > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > > > > Storage、S3、OSS up to now
> > > > > > * API:
> > > > > >     * Introduce some new restful API(experimental), including
> > > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > > schedule、task relation
> > > > > >     * Can get task instance id when triggering workflow via API
> > > > > >     * Registry: Introduce etcd, jdbc supported
> > > > > >
> > > > > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > > >
> > > > > > --
> > > > > > Best Wish
> > > > > > —Jay Chung
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Hi all,

Version 3.2.0 is nearly close and released during those days, but we
can not upload it to Apache SVN because too big for our tarball, We
created a Jira ticket to bump the svn limit for dolphinscheduler and
it is still in discussion. I will follow up when it has a further
update

On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zh...@gmail.com> wrote:
>
> Hi all,
>
> I had cherry-picked some recent commits to the branch 3.2.0-prepare,
> anyone who interested in checking 3.2.0 features could clone the
> latest branch to have a look
>
> On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zh...@gmail.com> wrote:
> >
> > Hi guys,
> >
> > I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
> > we introduce the test task and test datasource. It is a good feature
> > to allow user run in test environment.
> >
> > But I find out we have to create test datasource before create the
> > prod one. I think it make user confuse especial some one who do not
> > use test environment. So I think we should remove it in version 3.2.0.
> >
> > If you have different idea, please let me know in this mail thread
> >
> > [1]: https://github.com/apache/dolphinscheduler/pull/14381
> > [2]: https://github.com/apache/dolphinscheduler/pull/11670
> >
> > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
> > >
> > > Hi Community,
> > >
> > > I added a checklist for the version 3.2.0 release, everyone who
> > > interested in the incoming version can participate in our test, the
> > > detail checklist issue is in
> > > https://github.com/apache/dolphinscheduler/issues/14372
> > >
> > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> > > >
> > > > Thanks Jay.
> > > >
> > > > Looking forward to the 3.2.0 release.
> > > >
> > > > --
> > > > Best Wish
> > > > — Kerwin (zhuangchong)
> > > >
> > > >
> > > >
> > > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > > > release in July 2023 or August 2023.
> > > > >
> > > > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > > > it[1], including all features merged to branch dev after we release
> > > > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > > > additions and code base refactoring.
> > > > >
> > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > > > to Dolphinscheduler or want to contribute to the latest version is
> > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > > > start the standalone server, or build tarball for cluster mode test
> > > > > based on your wish. In this version, we need to test the core function
> > > > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > > > them. we allow need to test the resource center carefully because it
> > > > > changes a lot in this version.
> > > > >
> > > > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > > > a specific version for this incoming version name `3.2.0-prepare` in
> > > > > the bugs reported template.
> > > > >
> > > > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > > > you
> > > > >
> > > > > Some of the functional features list for 3.2.0:
> > > > >
> > > > > * User friendly
> > > > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > > > width
> > > > >     * Default tenant: Out of box run workflow by default tenant
> > > > >     * Run or rerun task in task instance
> > > > >     * Human readable json export result
> > > > > * Cloud native
> > > > >     * Kubernetes Real-time log and custom label support
> > > > >     * Support KEDA autoscaler worker
> > > > >     * Support Terraform deploy to AWS
> > > > >     * Multi arch for zookeper and postgresql(all services are supported
> > > > > now)
> > > > > * Resource center
> > > > >     * Refactor resource center, we use provider API to list and manage
> > > > > resource now
> > > > >     * Allow resource re-upload now
> > > > > * Datasource
> > > > >     * New datasource:
> > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > > Athena
> > > > >     * Support access key for redshift datasource
> > > > > * Tasks
> > > > >     * New tasks: azure datafactory、apache
> > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > > > >     * Improve for exists: Add presto for datax, parameter passing for
> > > > > python and http, sql task multiple sql delimiter, dependent depend
> > > > > itself
> > > > >     * Introduce task cache for MLops or other long-run tasks
> > > > > * Parameter
> > > > >     * Add new rule of build-in function
> > > > >     * Introduce file type parameter
> > > > >     * Startup parameters have the highest priority
> > > > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > > > Storage、S3、OSS up to now
> > > > > * API:
> > > > >     * Introduce some new restful API(experimental), including
> > > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > > schedule、task relation
> > > > >     * Can get task instance id when triggering workflow via API
> > > > >     * Registry: Introduce etcd, jdbc supported
> > > > >
> > > > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > > > >
> > > > > --
> > > > > Best Wish
> > > > > —Jay Chung
> > > > >
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Hi all,

I had cherry-picked some recent commits to the branch 3.2.0-prepare,
anyone who interested in checking 3.2.0 features could clone the
latest branch to have a look

On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zh...@gmail.com> wrote:
>
> Hi guys,
>
> I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
> we introduce the test task and test datasource. It is a good feature
> to allow user run in test environment.
>
> But I find out we have to create test datasource before create the
> prod one. I think it make user confuse especial some one who do not
> use test environment. So I think we should remove it in version 3.2.0.
>
> If you have different idea, please let me know in this mail thread
>
> [1]: https://github.com/apache/dolphinscheduler/pull/14381
> [2]: https://github.com/apache/dolphinscheduler/pull/11670
>
> On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
> >
> > Hi Community,
> >
> > I added a checklist for the version 3.2.0 release, everyone who
> > interested in the incoming version can participate in our test, the
> > detail checklist issue is in
> > https://github.com/apache/dolphinscheduler/issues/14372
> >
> > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> > >
> > > Thanks Jay.
> > >
> > > Looking forward to the 3.2.0 release.
> > >
> > > --
> > > Best Wish
> > > — Kerwin (zhuangchong)
> > >
> > >
> > >
> > > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> > >
> > > > Hi all,
> > > >
> > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > > release in July 2023 or August 2023.
> > > >
> > > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > > it[1], including all features merged to branch dev after we release
> > > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > > additions and code base refactoring.
> > > >
> > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > > to Dolphinscheduler or want to contribute to the latest version is
> > > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > > start the standalone server, or build tarball for cluster mode test
> > > > based on your wish. In this version, we need to test the core function
> > > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > > them. we allow need to test the resource center carefully because it
> > > > changes a lot in this version.
> > > >
> > > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > > a specific version for this incoming version name `3.2.0-prepare` in
> > > > the bugs reported template.
> > > >
> > > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > > you
> > > >
> > > > Some of the functional features list for 3.2.0:
> > > >
> > > > * User friendly
> > > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > > width
> > > >     * Default tenant: Out of box run workflow by default tenant
> > > >     * Run or rerun task in task instance
> > > >     * Human readable json export result
> > > > * Cloud native
> > > >     * Kubernetes Real-time log and custom label support
> > > >     * Support KEDA autoscaler worker
> > > >     * Support Terraform deploy to AWS
> > > >     * Multi arch for zookeper and postgresql(all services are supported
> > > > now)
> > > > * Resource center
> > > >     * Refactor resource center, we use provider API to list and manage
> > > > resource now
> > > >     * Allow resource re-upload now
> > > > * Datasource
> > > >     * New datasource:
> > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > > Athena
> > > >     * Support access key for redshift datasource
> > > > * Tasks
> > > >     * New tasks: azure datafactory、apache
> > > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > > >     * Improve for exists: Add presto for datax, parameter passing for
> > > > python and http, sql task multiple sql delimiter, dependent depend
> > > > itself
> > > >     * Introduce task cache for MLops or other long-run tasks
> > > > * Parameter
> > > >     * Add new rule of build-in function
> > > >     * Introduce file type parameter
> > > >     * Startup parameters have the highest priority
> > > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > > Storage、S3、OSS up to now
> > > > * API:
> > > >     * Introduce some new restful API(experimental), including
> > > > taskInstance、workflow state、workflowInstance、workflow and
> > > > schedule、task relation
> > > >     * Can get task instance id when triggering workflow via API
> > > >     * Registry: Introduce etcd, jdbc supported
> > > >
> > > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > > >
> > > > --
> > > > Best Wish
> > > > —Jay Chung
> > > >
> >
> >
> >
> > --
> > Best Wish
> > —Jay Chung
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Hi guys,

I submit new PR[1] to remove test datasource in version 3.2.0, in [2]
we introduce the test task and test datasource. It is a good feature
to allow user run in test environment.

But I find out we have to create test datasource before create the
prod one. I think it make user confuse especial some one who do not
use test environment. So I think we should remove it in version 3.2.0.

If you have different idea, please let me know in this mail thread

[1]: https://github.com/apache/dolphinscheduler/pull/14381
[2]: https://github.com/apache/dolphinscheduler/pull/11670

On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zh...@gmail.com> wrote:
>
> Hi Community,
>
> I added a checklist for the version 3.2.0 release, everyone who
> interested in the incoming version can participate in our test, the
> detail checklist issue is in
> https://github.com/apache/dolphinscheduler/issues/14372
>
> On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
> >
> > Thanks Jay.
> >
> > Looking forward to the 3.2.0 release.
> >
> > --
> > Best Wish
> > — Kerwin (zhuangchong)
> >
> >
> >
> > Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
> >
> > > Hi all,
> > >
> > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > > release in July 2023 or August 2023.
> > >
> > > Actually, I already created a new branch named `3.2.0-prepare` for
> > > it[1], including all features merged to branch dev after we release
> > > 3.1.0(date Sep 28, 2022), including many interesting functional
> > > additions and code base refactoring.
> > >
> > > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > > to Dolphinscheduler or want to contribute to the latest version is
> > > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > > start the standalone server, or build tarball for cluster mode test
> > > based on your wish. In this version, we need to test the core function
> > > of dolphinscheduler like CURD of project, workflow, task and execute
> > > them. we allow need to test the resource center carefully because it
> > > changes a lot in this version.
> > >
> > > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > > a specific version for this incoming version name `3.2.0-prepare` in
> > > the bugs reported template.
> > >
> > > And feel free to submit PR to fix bugs, whatever is reported by others or
> > > you
> > >
> > > Some of the functional features list for 3.2.0:
> > >
> > > * User friendly
> > >     * UI only: More hyperlinks jump between resource, adjustable column
> > > width
> > >     * Default tenant: Out of box run workflow by default tenant
> > >     * Run or rerun task in task instance
> > >     * Human readable json export result
> > > * Cloud native
> > >     * Kubernetes Real-time log and custom label support
> > >     * Support KEDA autoscaler worker
> > >     * Support Terraform deploy to AWS
> > >     * Multi arch for zookeper and postgresql(all services are supported
> > > now)
> > > * Resource center
> > >     * Refactor resource center, we use provider API to list and manage
> > > resource now
> > >     * Allow resource re-upload now
> > > * Datasource
> > >     * New datasource:
> > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > > Athena
> > >     * Support access key for redshift datasource
> > > * Tasks
> > >     * New tasks: azure datafactory、apache
> > > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> > >     * Improve for exists: Add presto for datax, parameter passing for
> > > python and http, sql task multiple sql delimiter, dependent depend
> > > itself
> > >     * Introduce task cache for MLops or other long-run tasks
> > > * Parameter
> > >     * Add new rule of build-in function
> > >     * Introduce file type parameter
> > >     * Startup parameters have the highest priority
> > > * Logs: Introduce remote logs mechanism and support Google Cloud
> > > Storage、S3、OSS up to now
> > > * API:
> > >     * Introduce some new restful API(experimental), including
> > > taskInstance、workflow state、workflowInstance、workflow and
> > > schedule、task relation
> > >     * Can get task instance id when triggering workflow via API
> > >     * Registry: Introduce etcd, jdbc supported
> > >
> > > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> > >
> > > --
> > > Best Wish
> > > —Jay Chung
> > >
>
>
>
> --
> Best Wish
> —Jay Chung



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Jay Chung <zh...@gmail.com>.

Hi Community,

I added a checklist for the version 3.2.0 release, everyone who
interested in the incoming version can participate in our test, the
detail checklist issue is in
https://github.com/apache/dolphinscheduler/issues/14372

On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang <zh...@gmail.com> wrote:
>
> Thanks Jay.
>
> Looking forward to the 3.2.0 release.
>
> --
> Best Wish
> — Kerwin (zhuangchong)
>
>
>
> Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：
>
> > Hi all,
> >
> > I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> > release in July 2023 or August 2023.
> >
> > Actually, I already created a new branch named `3.2.0-prepare` for
> > it[1], including all features merged to branch dev after we release
> > 3.1.0(date Sep 28, 2022), including many interesting functional
> > additions and code base refactoring.
> >
> > Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> > to Dolphinscheduler or want to contribute to the latest version is
> > welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> > start the standalone server, or build tarball for cluster mode test
> > based on your wish. In this version, we need to test the core function
> > of dolphinscheduler like CURD of project, workflow, task and execute
> > them. we allow need to test the resource center carefully because it
> > changes a lot in this version.
> >
> > You can report dolpinscheduler issue[2] if you find any bugs. We have
> > a specific version for this incoming version name `3.2.0-prepare` in
> > the bugs reported template.
> >
> > And feel free to submit PR to fix bugs, whatever is reported by others or
> > you
> >
> > Some of the functional features list for 3.2.0:
> >
> > * User friendly
> >     * UI only: More hyperlinks jump between resource, adjustable column
> > width
> >     * Default tenant: Out of box run workflow by default tenant
> >     * Run or rerun task in task instance
> >     * Human readable json export result
> > * Cloud native
> >     * Kubernetes Real-time log and custom label support
> >     * Support KEDA autoscaler worker
> >     * Support Terraform deploy to AWS
> >     * Multi arch for zookeper and postgresql(all services are supported
> > now)
> > * Resource center
> >     * Refactor resource center, we use provider API to list and manage
> > resource now
> >     * Allow resource re-upload now
> > * Datasource
> >     * New datasource:
> > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> > Athena
> >     * Support access key for redshift datasource
> > * Tasks
> >     * New tasks: azure datafactory、apache
> > linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
> >     * Improve for exists: Add presto for datax, parameter passing for
> > python and http, sql task multiple sql delimiter, dependent depend
> > itself
> >     * Introduce task cache for MLops or other long-run tasks
> > * Parameter
> >     * Add new rule of build-in function
> >     * Introduce file type parameter
> >     * Startup parameters have the highest priority
> > * Logs: Introduce remote logs mechanism and support Google Cloud
> > Storage、S3、OSS up to now
> > * API:
> >     * Introduce some new restful API(experimental), including
> > taskInstance、workflow state、workflowInstance、workflow and
> > schedule、task relation
> >     * Can get task instance id when triggering workflow via API
> >     * Registry: Introduce etcd, jdbc supported
> >
> > [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> > [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
> >
> > --
> > Best Wish
> > —Jay Chung
> >



-- 
Best Wish
—Jay Chung

Re: [DISCUSS] Version 3.2.0 release and help wanted for testing

Posted by Kerwin Zhuang <zh...@gmail.com>.

Thanks Jay.

Looking forward to the 3.2.0 release.

--
Best Wish
— Kerwin (zhuangchong)



Jay Chung <zh...@apache.org> 于2023年6月3日周六 20:01写道：

> Hi all,
>
> I am here to propose Apache Dolphinscheduler version 3.2.0 plan to
> release in July 2023 or August 2023.
>
> Actually, I already created a new branch named `3.2.0-prepare` for
> it[1], including all features merged to branch dev after we release
> 3.1.0(date Sep 28, 2022), including many interesting functional
> additions and code base refactoring.
>
> Version 3.2.0 is WIP in testing and bug fixing now, anyone interested
> to Dolphinscheduler or want to contribute to the latest version is
> welcome to test it. You can checkout the 3.2.0-prepare[1] branch and
> start the standalone server, or build tarball for cluster mode test
> based on your wish. In this version, we need to test the core function
> of dolphinscheduler like CURD of project, workflow, task and execute
> them. we allow need to test the resource center carefully because it
> changes a lot in this version.
>
> You can report dolpinscheduler issue[2] if you find any bugs. We have
> a specific version for this incoming version name `3.2.0-prepare` in
> the bugs reported template.
>
> And feel free to submit PR to fix bugs, whatever is reported by others or
> you
>
> Some of the functional features list for 3.2.0:
>
> * User friendly
>     * UI only: More hyperlinks jump between resource, adjustable column
> width
>     * Default tenant: Out of box run workflow by default tenant
>     * Run or rerun task in task instance
>     * Human readable json export result
> * Cloud native
>     * Kubernetes Real-time log and custom label support
>     * Support KEDA autoscaler worker
>     * Support Terraform deploy to AWS
>     * Multi arch for zookeper and postgresql(all services are supported
> now)
> * Resource center
>     * Refactor resource center, we use provider API to list and manage
> resource now
>     * Allow resource re-upload now
> * Datasource
>     * New datasource:
> snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS
> Athena
>     * Support access key for redshift datasource
> * Tasks
>     * New tasks: azure datafactory、apache
> linkis、Kubeflow、datasync、remote-shell、AWS Database Migration、Java Task
>     * Improve for exists: Add presto for datax, parameter passing for
> python and http, sql task multiple sql delimiter, dependent depend
> itself
>     * Introduce task cache for MLops or other long-run tasks
> * Parameter
>     * Add new rule of build-in function
>     * Introduce file type parameter
>     * Startup parameters have the highest priority
> * Logs: Introduce remote logs mechanism and support Google Cloud
> Storage、S3、OSS up to now
> * API:
>     * Introduce some new restful API(experimental), including
> taskInstance、workflow state、workflowInstance、workflow and
> schedule、task relation
>     * Can get task instance id when triggering workflow via API
>     * Registry: Introduce etcd, jdbc supported
>
> [1]: https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare
> [2]: https://github.com/apache/dolphinscheduler/issues/new/choose
>
> --
> Best Wish
> —Jay Chung
>