You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by Jiajie Zhong <zh...@gmail.com> on 2022/05/06 03:56:59 UTC
Re: Support AI task and the open source project about MLops
Sound good! and thanks Jieguang brings this up. DolphinScheduler
focuses on task scheduling and workflow orchestration, I think machine
learning is also another kind of orchestration, and most of the
machine learning source data or training samples are from data
warehouses or data lakes, which we already supported in the current
version. If we DolphinScheduler could support machine learning tasks
then users could finish their jobs in one single tool instead of
separately.
On Mon, Apr 25, 2022 at 2:39 PM 周哈哈 <ji...@163.com> wrote:
>
> Hi, community
>
> I have seen a Machine Learning Platform post on Medium. The post talk about Lizhi Machine Learning Platform&Apache DolphinScheduler.
> https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af <https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af>
>
>
> I want to use the dolphinscheduler to orchestrate some popular open-source projects about MLops, such as MLflow, Kubeflow, BentoML, Seldon Core, ClearML, TonY, etc. I think that can help users to build their custom MLops systems easier.
>
> For example, users can choose what deployment tools they like, or what training tools they like.
>
> Is somebody interested in the AI task about DS?
>
> I created an issue about the above: https://github.com/apache/dolphinscheduler/issues/9725 <https://github.com/apache/dolphinscheduler/issues/9725>
>
>
> Anyone who is interested is welcome to join us.
>
>
> Best Wish
> — Jieguang
--
Best Wish
— Jiajie
Re: Support AI task and the open source project about MLops
Posted by Jiajie Zhong <zh...@gmail.com>.
A new repository https://github.com/apache/dolphinscheduler-mlflow had
been created to hold some ml related resource, such as our example and
build-in machine learning algorithm
On Tue, May 17, 2022 at 10:19 AM Jiajie Zhong <zh...@gmail.com> wrote:
>
> The related discussion it is from
> https://github.com/apache/dolphinscheduler/pull/10036#discussion_r873164232,
> and jieguang want to have some build-in mlfow example and currently
> use his repository to hold all this code, I think the best way is we
> contribute some example to mlflow repository but it take time. So it
> is ok to create a new repository to hold this code as a subproject for
> dolphinscheduler, and I will create it.
>
> On Mon, May 16, 2022 at 4:27 PM 周哈哈 <ji...@163.com> wrote:
> >
> > Hi, I am supporting the MLflow task plugin to run the MLflow project. I have submitted the relevant PR (https://github.com/apache/dolphinscheduler/pull/10036).
> >
> > But I used a personal repository to provide algorithmic power out of the box, and hopefully a public repository.
> >
> > In order for engineers of different backgrounds to better use this task plugin, I suggest that:
> >
> > Create a repository for out-of-the-box machine learning ability.
> > Support for setting up directories to use users' own projects.
> >
> > So, can we create a repository for the first point?
> >
> >
> >
> >
> > > 2022年5月6日 上午11:56，Jiajie Zhong <zh...@gmail.com> 写道：
> > >
> > > Sound good! and thanks Jieguang brings this up. DolphinScheduler
> > > focuses on task scheduling and workflow orchestration, I think machine
> > > learning is also another kind of orchestration, and most of the
> > > machine learning source data or training samples are from data
> > > warehouses or data lakes, which we already supported in the current
> > > version. If we DolphinScheduler could support machine learning tasks
> > > then users could finish their jobs in one single tool instead of
> > > separately.
> > >
> > > On Mon, Apr 25, 2022 at 2:39 PM 周哈哈 <ji...@163.com> wrote:
> > >>
> > >> Hi, community
> > >>
> > >> I have seen a Machine Learning Platform post on Medium. The post talk about Lizhi Machine Learning Platform&Apache DolphinScheduler.
> > >> https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af <https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af>
> > >>
> > >>
> > >> I want to use the dolphinscheduler to orchestrate some popular open-source projects about MLops, such as MLflow, Kubeflow, BentoML, Seldon Core, ClearML, TonY, etc. I think that can help users to build their custom MLops systems easier.
> > >>
> > >> For example, users can choose what deployment tools they like, or what training tools they like.
> > >>
> > >> Is somebody interested in the AI task about DS?
> > >>
> > >> I created an issue about the above: https://github.com/apache/dolphinscheduler/issues/9725 <https://github.com/apache/dolphinscheduler/issues/9725>
> > >>
> > >>
> > >> Anyone who is interested is welcome to join us.
> > >>
> > >>
> > >> Best Wish
> > >> — Jieguang
> > >
> > >
> > >
> > > --
> > > Best Wish
> > > — Jiajie
> >
>
>
> --
> Best Wish
> — Jiajie
--
Best Wish
— Jiajie
Re: Support AI task and the open source project about MLops
Posted by Jiajie Zhong <zh...@gmail.com>.
The related discussion it is from
https://github.com/apache/dolphinscheduler/pull/10036#discussion_r873164232,
and jieguang want to have some build-in mlfow example and currently
use his repository to hold all this code, I think the best way is we
contribute some example to mlflow repository but it take time. So it
is ok to create a new repository to hold this code as a subproject for
dolphinscheduler, and I will create it.
On Mon, May 16, 2022 at 4:27 PM 周哈哈 <ji...@163.com> wrote:
>
> Hi, I am supporting the MLflow task plugin to run the MLflow project. I have submitted the relevant PR (https://github.com/apache/dolphinscheduler/pull/10036).
>
> But I used a personal repository to provide algorithmic power out of the box, and hopefully a public repository.
>
> In order for engineers of different backgrounds to better use this task plugin, I suggest that:
>
> Create a repository for out-of-the-box machine learning ability.
> Support for setting up directories to use users' own projects.
>
> So, can we create a repository for the first point?
>
>
>
>
> > 2022年5月6日 上午11:56，Jiajie Zhong <zh...@gmail.com> 写道：
> >
> > Sound good! and thanks Jieguang brings this up. DolphinScheduler
> > focuses on task scheduling and workflow orchestration, I think machine
> > learning is also another kind of orchestration, and most of the
> > machine learning source data or training samples are from data
> > warehouses or data lakes, which we already supported in the current
> > version. If we DolphinScheduler could support machine learning tasks
> > then users could finish their jobs in one single tool instead of
> > separately.
> >
> > On Mon, Apr 25, 2022 at 2:39 PM 周哈哈 <ji...@163.com> wrote:
> >>
> >> Hi, community
> >>
> >> I have seen a Machine Learning Platform post on Medium. The post talk about Lizhi Machine Learning Platform&Apache DolphinScheduler.
> >> https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af <https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af>
> >>
> >>
> >> I want to use the dolphinscheduler to orchestrate some popular open-source projects about MLops, such as MLflow, Kubeflow, BentoML, Seldon Core, ClearML, TonY, etc. I think that can help users to build their custom MLops systems easier.
> >>
> >> For example, users can choose what deployment tools they like, or what training tools they like.
> >>
> >> Is somebody interested in the AI task about DS?
> >>
> >> I created an issue about the above: https://github.com/apache/dolphinscheduler/issues/9725 <https://github.com/apache/dolphinscheduler/issues/9725>
> >>
> >>
> >> Anyone who is interested is welcome to join us.
> >>
> >>
> >> Best Wish
> >> — Jieguang
> >
> >
> >
> > --
> > Best Wish
> > — Jiajie
>
--
Best Wish
— Jiajie
Re: Support AI task and the open source project about MLops
Posted by 周哈哈 <ji...@163.com>.
Hi, I am supporting the MLflow task plugin to run the MLflow project. I have submitted the relevant PR (https://github.com/apache/dolphinscheduler/pull/10036).
But I used a personal repository to provide algorithmic power out of the box, and hopefully a public repository.
In order for engineers of different backgrounds to better use this task plugin, I suggest that:
Create a repository for out-of-the-box machine learning ability.
Support for setting up directories to use users' own projects.
So, can we create a repository for the first point?
> 2022年5月6日 上午11:56，Jiajie Zhong <zh...@gmail.com> 写道：
>
> Sound good! and thanks Jieguang brings this up. DolphinScheduler
> focuses on task scheduling and workflow orchestration, I think machine
> learning is also another kind of orchestration, and most of the
> machine learning source data or training samples are from data
> warehouses or data lakes, which we already supported in the current
> version. If we DolphinScheduler could support machine learning tasks
> then users could finish their jobs in one single tool instead of
> separately.
>
> On Mon, Apr 25, 2022 at 2:39 PM 周哈哈 <ji...@163.com> wrote:
>>
>> Hi, community
>>
>> I have seen a Machine Learning Platform post on Medium. The post talk about Lizhi Machine Learning Platform&Apache DolphinScheduler.
>> https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af <https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af>
>>
>>
>> I want to use the dolphinscheduler to orchestrate some popular open-source projects about MLops, such as MLflow, Kubeflow, BentoML, Seldon Core, ClearML, TonY, etc. I think that can help users to build their custom MLops systems easier.
>>
>> For example, users can choose what deployment tools they like, or what training tools they like.
>>
>> Is somebody interested in the AI task about DS?
>>
>> I created an issue about the above: https://github.com/apache/dolphinscheduler/issues/9725 <https://github.com/apache/dolphinscheduler/issues/9725>
>>
>>
>> Anyone who is interested is welcome to join us.
>>
>>
>> Best Wish
>> — Jieguang
>
>
>
> --
> Best Wish
> — Jiajie