Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

Hi all,

This is Xun Liu contributing to the Submarine project for deep learning
workloads running with big data workloads together on Hadoop clusters.

There are a bunch of integrations of Submarine to other projects are
finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
of Submarine is going to integrate with more projects like Apache Arrow,
Redis, MLflow, etc. & be able to handle end-to-end machine learning use
cases like model serving, notebook management, advanced training
optimizations (like auto parameter tuning, memory cache optimizations for
large datasets for training, etc.), and make it run on other platforms like
Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
to Apache so we can put Submarine and TonY together to the same codebase
(Page #30.
https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
).

This expands the scope of the original Submarine project in exciting new
ways. Toward that end, would it make sense to create a separate Submarine
project at Apache? This can make faster adoption of Submarine, and allow
Submarine to grow to a full-blown machine learning platform.

There will be lots of technical details to work out, but any initial
thoughts on this?

Best Regards,
Xun Liu

Re: Re: Any thoughts making Submarine a separate Apache project?

[赵欣 <xi...@seu.edu.cn>]

Hi

I am a teacher at Southeast University (https://www.seu.edu.cn/). We are a major in electrical engineering. Our teaching teams and students use bigoop submarine for big data analysis and automation control of electrical equipment.

Many thanks to the hadoop community for providing us with machine learning tools like submarine.

I wish hadoop submarine is getting better and better.
 
==============================
赵欣
东南大学电气工程学院
-----------------------------------------------------
Zhao XIN
School of Electrical Engineering
==============================
2019-07-22
 
From: Xun Liu
Date: 2019-07-22 16:06
To: 赵欣; adam.antal
CC: common-dev; yarn-dev; hdfs-dev; mapreduce-dev; submarine-dev; user
Subject: Re: Any thoughts making Submarine a separate Apache project?
@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
Hi

I am a teacher at Southeast University (https://www.seu.edu.cn/). We are a major in electrical engineering. Our teaching teams and students use bigoop submarine for big data analysis and automation control of electrical equipment.

Many thanks to the hadoop community for providing us with machine learning tools like submarine.

I wish hadoop submarine is getting better and better.

 
==============================
赵欣
东南大学电气工程学院
-----------------------------------------------------
Zhao XIN
School of Electrical Engineering
==============================
2019-07-18
 
From: Xun Liu
Date: 2019-07-18 09:46
To: xinzhao
Subject: Fwd: Re: Any thoughts making Submarine a separate Apache project?


---------- Forwarded message ---------
发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
Date: 2019年7月17日周三 下午3:17
Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <ru...@gmail.com>
Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>, yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <hd...@hadoop.apache.org>, mapreduce-dev <ma...@hadoop.apache.org>, submarine-dev <su...@hadoop.apache.org>


+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli <vi...@apache.org> could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan <wh...@gmail.com> wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>>> >>> The submarine development team has completed the following
>>> preparations:
>>> >>> 1. Established a temporary test repository on Github.
>>> >>> 2. Change the package name of hadoop submarine from
>>> org.hadoop.submarine
>>> >> to
>>> >>> org.submarine
>>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> >>> 4. On the Github docked travis-ci system, all test cases have been
>>> >> tested;
>>> >>> 5. Several Hadoop submarine users completed the system test using the
>>> >> code
>>> >>> in this repository.
>>> >>>
>>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> >>>
>>> >>>> Hi
>>> >>>>
>>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
>>> We
>>> >> are
>>> >>>> a major in electrical engineering. Our teaching teams and students
>>> use
>>> >>>> bigoop submarine for big data analysis and automation control of
>>> >>> electrical
>>> >>>> equipment.
>>> >>>>
>>> >>>> Many thanks to the hadoop community for providing us with machine
>>> >>> learning
>>> >>>> tools like submarine.
>>> >>>>
>>> >>>> I wish hadoop submarine is getting better and better.
>>> >>>>
>>> >>>>
>>> >>>> ==============================
>>> >>>> 赵欣
>>> >>>> 东南大学电气工程学院
>>> >>>>
>>> >>>> -----------------------------------------------------
>>> >>>>
>>> >>>> Zhao XIN
>>> >>>>
>>> >>>> School of Electrical Engineering
>>> >>>>
>>> >>>> ==============================
>>> >>>> 2019-07-18
>>> >>>>
>>> >>>>
>>> >>>> *From:* Xun Liu <li...@apache.org>
>>> >>>> *Date:* 2019-07-18 09:46
>>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>> >>>> project?
>>> >>>>
>>> >>>>
>>> >>>> ---------- Forwarded message ---------
>>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>> >>>> Date: 2019年7月17日周三 下午3:17
>>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>>> >> project?
>>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>> >>>> runlin512@gmail.com>
>>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> >>> common-dev@hadoop.apache.org>,
>>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>> >>>> submarine-dev@hadoop.apache.org>
>>> >>>>
>>> >>>>
>>> >>>> +1 ，Good idea, we are very much looking forward to it.
>>> >>>>
>>> >>>> ------------------------------
>>> >>>> dashuiguailuyun@gmail.com
>>> >>>>
>>> >>>>
>>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>> >>>> *Date:* 2019-07-17 14:55
>>> >>>> *To:* runlin zhang <ru...@gmail.com>
>>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>> >>>> <co...@hadoop.apache.org>; yarn-dev <
>>> yarn-dev@hadoop.apache.org>;
>>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>>> >>>> <su...@hadoop.apache.org>
>>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>>> project?
>>> >>>> +1, this is a very great idea.
>>> >>>> As Hadoop repository has already grown huge and contains many
>>> >> projects, I
>>> >>>> think in general it's a good idea to separate projects in the early
>>> >>> phase.
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> +1 ，That will be great ！
>>> >>>>>
>>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> >>> learning
>>> >>>>>> workloads running with big data workloads together on Hadoop
>>> >>> clusters.
>>> >>>>>>
>>> >>>>>> There are a bunch of integrations of Submarine to other projects
>>> >> are
>>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> >>> next
>>> >>>>> step
>>> >>>>>> of Submarine is going to integrate with more projects like Apache
>>> >>>> Arrow,
>>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
>>> learning
>>> >>> use
>>> >>>>>> cases like model serving, notebook management, advanced training
>>> >>>>>> optimizations (like auto parameter tuning, memory cache
>>> >> optimizations
>>> >>>> for
>>> >>>>>> large datasets for training, etc.), and make it run on other
>>> >>> platforms
>>> >>>>> like
>>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate
>>> TonY
>>> >>>>> project
>>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>>> >>>> codebase
>>> >>>>>> (Page #30.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>> >>>>>> ).
>>> >>>>>>
>>> >>>>>> This expands the scope of the original Submarine project in
>>> >> exciting
>>> >>>> new
>>> >>>>>> ways. Toward that end, would it make sense to create a separate
>>> >>>> Submarine
>>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>>> >>>> allow
>>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>>> >>>>>>
>>> >>>>>> There will be lots of technical details to work out, but any
>>> >> initial
>>> >>>>>> thoughts on this?
>>> >>>>>>
>>> >>>>>> Best Regards,
>>> >>>>>> Xun Liu
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli <vi...@apache.org> could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan <wh...@gmail.com> wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>>> >>> The submarine development team has completed the following
>>> preparations:
>>> >>> 1. Established a temporary test repository on Github.
>>> >>> 2. Change the package name of hadoop submarine from
>>> org.hadoop.submarine
>>> >> to
>>> >>> org.submarine
>>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> >>> 4. On the Github docked travis-ci system, all test cases have been
>>> >> tested;
>>> >>> 5. Several Hadoop submarine users completed the system test using the
>>> >> code
>>> >>> in this repository.
>>> >>>
>>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> >>>
>>> >>>> Hi
>>> >>>>
>>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
>>> We
>>> >> are
>>> >>>> a major in electrical engineering. Our teaching teams and students
>>> use
>>> >>>> bigoop submarine for big data analysis and automation control of
>>> >>> electrical
>>> >>>> equipment.
>>> >>>>
>>> >>>> Many thanks to the hadoop community for providing us with machine
>>> >>> learning
>>> >>>> tools like submarine.
>>> >>>>
>>> >>>> I wish hadoop submarine is getting better and better.
>>> >>>>
>>> >>>>
>>> >>>> ==============================
>>> >>>> 赵欣
>>> >>>> 东南大学电气工程学院
>>> >>>>
>>> >>>> -----------------------------------------------------
>>> >>>>
>>> >>>> Zhao XIN
>>> >>>>
>>> >>>> School of Electrical Engineering
>>> >>>>
>>> >>>> ==============================
>>> >>>> 2019-07-18
>>> >>>>
>>> >>>>
>>> >>>> *From:* Xun Liu <li...@apache.org>
>>> >>>> *Date:* 2019-07-18 09:46
>>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>> >>>> project?
>>> >>>>
>>> >>>>
>>> >>>> ---------- Forwarded message ---------
>>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>> >>>> Date: 2019年7月17日周三 下午3:17
>>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>>> >> project?
>>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>> >>>> runlin512@gmail.com>
>>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> >>> common-dev@hadoop.apache.org>,
>>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>> >>>> submarine-dev@hadoop.apache.org>
>>> >>>>
>>> >>>>
>>> >>>> +1 ，Good idea, we are very much looking forward to it.
>>> >>>>
>>> >>>> ------------------------------
>>> >>>> dashuiguailuyun@gmail.com
>>> >>>>
>>> >>>>
>>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>> >>>> *Date:* 2019-07-17 14:55
>>> >>>> *To:* runlin zhang <ru...@gmail.com>
>>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>> >>>> <co...@hadoop.apache.org>; yarn-dev <
>>> yarn-dev@hadoop.apache.org>;
>>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>>> >>>> <su...@hadoop.apache.org>
>>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>>> project?
>>> >>>> +1, this is a very great idea.
>>> >>>> As Hadoop repository has already grown huge and contains many
>>> >> projects, I
>>> >>>> think in general it's a good idea to separate projects in the early
>>> >>> phase.
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> +1 ，That will be great ！
>>> >>>>>
>>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> >>> learning
>>> >>>>>> workloads running with big data workloads together on Hadoop
>>> >>> clusters.
>>> >>>>>>
>>> >>>>>> There are a bunch of integrations of Submarine to other projects
>>> >> are
>>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> >>> next
>>> >>>>> step
>>> >>>>>> of Submarine is going to integrate with more projects like Apache
>>> >>>> Arrow,
>>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
>>> learning
>>> >>> use
>>> >>>>>> cases like model serving, notebook management, advanced training
>>> >>>>>> optimizations (like auto parameter tuning, memory cache
>>> >> optimizations
>>> >>>> for
>>> >>>>>> large datasets for training, etc.), and make it run on other
>>> >>> platforms
>>> >>>>> like
>>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate
>>> TonY
>>> >>>>> project
>>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>>> >>>> codebase
>>> >>>>>> (Page #30.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>> >>>>>> ).
>>> >>>>>>
>>> >>>>>> This expands the scope of the original Submarine project in
>>> >> exciting
>>> >>>> new
>>> >>>>>> ways. Toward that end, would it make sense to create a separate
>>> >>>> Submarine
>>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>>> >>>> allow
>>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>>> >>>>>>
>>> >>>>>> There will be lots of technical details to work out, but any
>>> >> initial
>>> >>>>>> thoughts on this?
>>> >>>>>>
>>> >>>>>> Best Regards,
>>> >>>>>> Xun Liu
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli <vi...@apache.org> could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan <wh...@gmail.com> wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>>> >>> The submarine development team has completed the following
>>> preparations:
>>> >>> 1. Established a temporary test repository on Github.
>>> >>> 2. Change the package name of hadoop submarine from
>>> org.hadoop.submarine
>>> >> to
>>> >>> org.submarine
>>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> >>> 4. On the Github docked travis-ci system, all test cases have been
>>> >> tested;
>>> >>> 5. Several Hadoop submarine users completed the system test using the
>>> >> code
>>> >>> in this repository.
>>> >>>
>>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> >>>
>>> >>>> Hi
>>> >>>>
>>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
>>> We
>>> >> are
>>> >>>> a major in electrical engineering. Our teaching teams and students
>>> use
>>> >>>> bigoop submarine for big data analysis and automation control of
>>> >>> electrical
>>> >>>> equipment.
>>> >>>>
>>> >>>> Many thanks to the hadoop community for providing us with machine
>>> >>> learning
>>> >>>> tools like submarine.
>>> >>>>
>>> >>>> I wish hadoop submarine is getting better and better.
>>> >>>>
>>> >>>>
>>> >>>> ==============================
>>> >>>> 赵欣
>>> >>>> 东南大学电气工程学院
>>> >>>>
>>> >>>> -----------------------------------------------------
>>> >>>>
>>> >>>> Zhao XIN
>>> >>>>
>>> >>>> School of Electrical Engineering
>>> >>>>
>>> >>>> ==============================
>>> >>>> 2019-07-18
>>> >>>>
>>> >>>>
>>> >>>> *From:* Xun Liu <li...@apache.org>
>>> >>>> *Date:* 2019-07-18 09:46
>>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>> >>>> project?
>>> >>>>
>>> >>>>
>>> >>>> ---------- Forwarded message ---------
>>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>> >>>> Date: 2019年7月17日周三 下午3:17
>>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>>> >> project?
>>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>> >>>> runlin512@gmail.com>
>>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> >>> common-dev@hadoop.apache.org>,
>>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>> >>>> submarine-dev@hadoop.apache.org>
>>> >>>>
>>> >>>>
>>> >>>> +1 ，Good idea, we are very much looking forward to it.
>>> >>>>
>>> >>>> ------------------------------
>>> >>>> dashuiguailuyun@gmail.com
>>> >>>>
>>> >>>>
>>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>> >>>> *Date:* 2019-07-17 14:55
>>> >>>> *To:* runlin zhang <ru...@gmail.com>
>>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>> >>>> <co...@hadoop.apache.org>; yarn-dev <
>>> yarn-dev@hadoop.apache.org>;
>>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>>> >>>> <su...@hadoop.apache.org>
>>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>>> project?
>>> >>>> +1, this is a very great idea.
>>> >>>> As Hadoop repository has already grown huge and contains many
>>> >> projects, I
>>> >>>> think in general it's a good idea to separate projects in the early
>>> >>> phase.
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> +1 ，That will be great ！
>>> >>>>>
>>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> >>> learning
>>> >>>>>> workloads running with big data workloads together on Hadoop
>>> >>> clusters.
>>> >>>>>>
>>> >>>>>> There are a bunch of integrations of Submarine to other projects
>>> >> are
>>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> >>> next
>>> >>>>> step
>>> >>>>>> of Submarine is going to integrate with more projects like Apache
>>> >>>> Arrow,
>>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
>>> learning
>>> >>> use
>>> >>>>>> cases like model serving, notebook management, advanced training
>>> >>>>>> optimizations (like auto parameter tuning, memory cache
>>> >> optimizations
>>> >>>> for
>>> >>>>>> large datasets for training, etc.), and make it run on other
>>> >>> platforms
>>> >>>>> like
>>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate
>>> TonY
>>> >>>>> project
>>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>>> >>>> codebase
>>> >>>>>> (Page #30.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>> >>>>>> ).
>>> >>>>>>
>>> >>>>>> This expands the scope of the original Submarine project in
>>> >> exciting
>>> >>>> new
>>> >>>>>> ways. Toward that end, would it make sense to create a separate
>>> >>>> Submarine
>>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>>> >>>> allow
>>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>>> >>>>>>
>>> >>>>>> There will be lots of technical details to work out, but any
>>> >> initial
>>> >>>>>> thoughts on this?
>>> >>>>>>
>>> >>>>>> Best Regards,
>>> >>>>>> Xun Liu
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli <vi...@apache.org> could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan <wh...@gmail.com> wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>>> >>> The submarine development team has completed the following
>>> preparations:
>>> >>> 1. Established a temporary test repository on Github.
>>> >>> 2. Change the package name of hadoop submarine from
>>> org.hadoop.submarine
>>> >> to
>>> >>> org.submarine
>>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> >>> 4. On the Github docked travis-ci system, all test cases have been
>>> >> tested;
>>> >>> 5. Several Hadoop submarine users completed the system test using the
>>> >> code
>>> >>> in this repository.
>>> >>>
>>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> >>>
>>> >>>> Hi
>>> >>>>
>>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
>>> We
>>> >> are
>>> >>>> a major in electrical engineering. Our teaching teams and students
>>> use
>>> >>>> bigoop submarine for big data analysis and automation control of
>>> >>> electrical
>>> >>>> equipment.
>>> >>>>
>>> >>>> Many thanks to the hadoop community for providing us with machine
>>> >>> learning
>>> >>>> tools like submarine.
>>> >>>>
>>> >>>> I wish hadoop submarine is getting better and better.
>>> >>>>
>>> >>>>
>>> >>>> ==============================
>>> >>>> 赵欣
>>> >>>> 东南大学电气工程学院
>>> >>>>
>>> >>>> -----------------------------------------------------
>>> >>>>
>>> >>>> Zhao XIN
>>> >>>>
>>> >>>> School of Electrical Engineering
>>> >>>>
>>> >>>> ==============================
>>> >>>> 2019-07-18
>>> >>>>
>>> >>>>
>>> >>>> *From:* Xun Liu <li...@apache.org>
>>> >>>> *Date:* 2019-07-18 09:46
>>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>> >>>> project?
>>> >>>>
>>> >>>>
>>> >>>> ---------- Forwarded message ---------
>>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>> >>>> Date: 2019年7月17日周三 下午3:17
>>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>>> >> project?
>>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>> >>>> runlin512@gmail.com>
>>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> >>> common-dev@hadoop.apache.org>,
>>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>> >>>> submarine-dev@hadoop.apache.org>
>>> >>>>
>>> >>>>
>>> >>>> +1 ，Good idea, we are very much looking forward to it.
>>> >>>>
>>> >>>> ------------------------------
>>> >>>> dashuiguailuyun@gmail.com
>>> >>>>
>>> >>>>
>>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>> >>>> *Date:* 2019-07-17 14:55
>>> >>>> *To:* runlin zhang <ru...@gmail.com>
>>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>> >>>> <co...@hadoop.apache.org>; yarn-dev <
>>> yarn-dev@hadoop.apache.org>;
>>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>>> >>>> <su...@hadoop.apache.org>
>>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>>> project?
>>> >>>> +1, this is a very great idea.
>>> >>>> As Hadoop repository has already grown huge and contains many
>>> >> projects, I
>>> >>>> think in general it's a good idea to separate projects in the early
>>> >>> phase.
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> +1 ，That will be great ！
>>> >>>>>
>>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> >>> learning
>>> >>>>>> workloads running with big data workloads together on Hadoop
>>> >>> clusters.
>>> >>>>>>
>>> >>>>>> There are a bunch of integrations of Submarine to other projects
>>> >> are
>>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> >>> next
>>> >>>>> step
>>> >>>>>> of Submarine is going to integrate with more projects like Apache
>>> >>>> Arrow,
>>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
>>> learning
>>> >>> use
>>> >>>>>> cases like model serving, notebook management, advanced training
>>> >>>>>> optimizations (like auto parameter tuning, memory cache
>>> >> optimizations
>>> >>>> for
>>> >>>>>> large datasets for training, etc.), and make it run on other
>>> >>> platforms
>>> >>>>> like
>>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate
>>> TonY
>>> >>>>> project
>>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>>> >>>> codebase
>>> >>>>>> (Page #30.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>> >>>>>> ).
>>> >>>>>>
>>> >>>>>> This expands the scope of the original Submarine project in
>>> >> exciting
>>> >>>> new
>>> >>>>>> ways. Toward that end, would it make sense to create a separate
>>> >>>> Submarine
>>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>>> >>>> allow
>>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>>> >>>>>>
>>> >>>>>> There will be lots of technical details to work out, but any
>>> >> initial
>>> >>>>>> thoughts on this?
>>> >>>>>>
>>> >>>>>> Best Regards,
>>> >>>>>> Xun Liu
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli <vi...@apache.org> could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan <wh...@gmail.com> wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>>> >>> The submarine development team has completed the following
>>> preparations:
>>> >>> 1. Established a temporary test repository on Github.
>>> >>> 2. Change the package name of hadoop submarine from
>>> org.hadoop.submarine
>>> >> to
>>> >>> org.submarine
>>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> >>> 4. On the Github docked travis-ci system, all test cases have been
>>> >> tested;
>>> >>> 5. Several Hadoop submarine users completed the system test using the
>>> >> code
>>> >>> in this repository.
>>> >>>
>>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> >>>
>>> >>>> Hi
>>> >>>>
>>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
>>> We
>>> >> are
>>> >>>> a major in electrical engineering. Our teaching teams and students
>>> use
>>> >>>> bigoop submarine for big data analysis and automation control of
>>> >>> electrical
>>> >>>> equipment.
>>> >>>>
>>> >>>> Many thanks to the hadoop community for providing us with machine
>>> >>> learning
>>> >>>> tools like submarine.
>>> >>>>
>>> >>>> I wish hadoop submarine is getting better and better.
>>> >>>>
>>> >>>>
>>> >>>> ==============================
>>> >>>> 赵欣
>>> >>>> 东南大学电气工程学院
>>> >>>>
>>> >>>> -----------------------------------------------------
>>> >>>>
>>> >>>> Zhao XIN
>>> >>>>
>>> >>>> School of Electrical Engineering
>>> >>>>
>>> >>>> ==============================
>>> >>>> 2019-07-18
>>> >>>>
>>> >>>>
>>> >>>> *From:* Xun Liu <li...@apache.org>
>>> >>>> *Date:* 2019-07-18 09:46
>>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>> >>>> project?
>>> >>>>
>>> >>>>
>>> >>>> ---------- Forwarded message ---------
>>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>> >>>> Date: 2019年7月17日周三 下午3:17
>>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>>> >> project?
>>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>> >>>> runlin512@gmail.com>
>>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> >>> common-dev@hadoop.apache.org>,
>>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>> >>>> submarine-dev@hadoop.apache.org>
>>> >>>>
>>> >>>>
>>> >>>> +1 ，Good idea, we are very much looking forward to it.
>>> >>>>
>>> >>>> ------------------------------
>>> >>>> dashuiguailuyun@gmail.com
>>> >>>>
>>> >>>>
>>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>> >>>> *Date:* 2019-07-17 14:55
>>> >>>> *To:* runlin zhang <ru...@gmail.com>
>>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>> >>>> <co...@hadoop.apache.org>; yarn-dev <
>>> yarn-dev@hadoop.apache.org>;
>>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>>> >>>> <su...@hadoop.apache.org>
>>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>>> project?
>>> >>>> +1, this is a very great idea.
>>> >>>> As Hadoop repository has already grown huge and contains many
>>> >> projects, I
>>> >>>> think in general it's a good idea to separate projects in the early
>>> >>> phase.
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> +1 ，That will be great ！
>>> >>>>>
>>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> >>> learning
>>> >>>>>> workloads running with big data workloads together on Hadoop
>>> >>> clusters.
>>> >>>>>>
>>> >>>>>> There are a bunch of integrations of Submarine to other projects
>>> >> are
>>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> >>> next
>>> >>>>> step
>>> >>>>>> of Submarine is going to integrate with more projects like Apache
>>> >>>> Arrow,
>>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
>>> learning
>>> >>> use
>>> >>>>>> cases like model serving, notebook management, advanced training
>>> >>>>>> optimizations (like auto parameter tuning, memory cache
>>> >> optimizations
>>> >>>> for
>>> >>>>>> large datasets for training, etc.), and make it run on other
>>> >>> platforms
>>> >>>>> like
>>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate
>>> TonY
>>> >>>>> project
>>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>>> >>>> codebase
>>> >>>>>> (Page #30.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>> >>>>>> ).
>>> >>>>>>
>>> >>>>>> This expands the scope of the original Submarine project in
>>> >> exciting
>>> >>>> new
>>> >>>>>> ways. Toward that end, would it make sense to create a separate
>>> >>>> Submarine
>>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>>> >>>> allow
>>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>>> >>>>>>
>>> >>>>>> There will be lots of technical details to work out, but any
>>> >> initial
>>> >>>>>> thoughts on this?
>>> >>>>>>
>>> >>>>>> Best Regards,
>>> >>>>>> Xun Liu
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <du...@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <li...@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xi...@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ，Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <ru...@gmail.com>
>> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>> >>>> <ma...@hadoop.apache.org>; submarine-dev
>> >>>> <su...@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ，That will be great ！
>> >>>>>
>> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine, and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[俊平堵 <du...@gmail.com>]

Thanks Vinod for these great suggestions. I agree most of your comments
above.
 "For the Apache Hadoop community, this will be treated simply as
code-change and so need a committer +1?". IIUC, this should be treated as
feature branch merge, so may be 3 committer +1 is needed here according to
https://hadoop.apache.org/bylaws.html?

bq. Can somebody who have cycles and been on the ASF lists for a while look
into the process here?
I can check with ASF members who has experience on this if no one haven't
yet.

Thanks,

Junping

Vinod Kumar Vavilapalli <vi...@apache.org> 于2019年7月29日周一 下午9:46写道：

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==============================
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -----------------------------------------------------
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==============================
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu <li...@apache.org>
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao <xi...@seu.edu.cn>
> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> >>>> project?
> >>>>
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> >>>> Date: 2019年7月17日周三 下午3:17
> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
> >> project?
> >>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> >>>> runlin512@gmail.com>
> >>>> Cc: Xun Liu <li...@apache.org>, common-dev <
> >>> common-dev@hadoop.apache.org>,
> >>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> >>>> submarine-dev@hadoop.apache.org>
> >>>>
> >>>>
> >>>> +1 ，Good idea, we are very much looking forward to it.
> >>>>
> >>>> ------------------------------
> >>>> dashuiguailuyun@gmail.com
> >>>>
> >>>>
> >>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> >>>> *Date:* 2019-07-17 14:55
> >>>> *To:* runlin zhang <ru...@gmail.com>
> >>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> >>>> <co...@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
> >;
> >>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> >>>> <ma...@hadoop.apache.org>; submarine-dev
> >>>> <su...@hadoop.apache.org>
> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
> project?
> >>>> +1, this is a very great idea.
> >>>> As Hadoop repository has already grown huge and contains many
> >> projects, I
> >>>> think in general it's a good idea to separate projects in the early
> >>> phase.
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >>>>
> >>>>> +1 ，That will be great ！
> >>>>>
> >>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> This is Xun Liu contributing to the Submarine project for deep
> >>> learning
> >>>>>> workloads running with big data workloads together on Hadoop
> >>> clusters.
> >>>>>>
> >>>>>> There are a bunch of integrations of Submarine to other projects
> >> are
> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> >>> next
> >>>>> step
> >>>>>> of Submarine is going to integrate with more projects like Apache
> >>>> Arrow,
> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
> >>> use
> >>>>>> cases like model serving, notebook management, advanced training
> >>>>>> optimizations (like auto parameter tuning, memory cache
> >> optimizations
> >>>> for
> >>>>>> large datasets for training, etc.), and make it run on other
> >>> platforms
> >>>>> like
> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> >>>>> project
> >>>>>> to Apache so we can put Submarine and TonY together to the same
> >>>> codebase
> >>>>>> (Page #30.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> >>>>>> ).
> >>>>>>
> >>>>>> This expands the scope of the original Submarine project in
> >> exciting
> >>>> new
> >>>>>> ways. Toward that end, would it make sense to create a separate
> >>>> Submarine
> >>>>>> project at Apache? This can make faster adoption of Submarine, and
> >>>> allow
> >>>>>> Submarine to grow to a full-blown machine learning platform.
> >>>>>>
> >>>>>> There will be lots of technical details to work out, but any
> >> initial
> >>>>>> thoughts on this?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Xun Liu
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this enthusiasm turns into building a real, independent but more importantly a sustainable community.

Given that there were two official releases off the Apache Hadoop project, I doubt if you'd need to go through the incubator process. Instead you can directly propose a new TLP at ASF board. The last few times this happened was with ORC, and long before that with Hive, HBase etc. Can somebody who have cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and so need a committer +1? You can be more gently by formally doing a vote once a process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can exercise their will to join this new project as PMC/Committers respectively without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan <wh...@gmail.com> wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>>> 
>>>> Hi
>>>> 
>>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
>>>> a major in electrical engineering. Our teaching teams and students use
>>>> bigoop submarine for big data analysis and automation control of
>>> electrical
>>>> equipment.
>>>> 
>>>> Many thanks to the hadoop community for providing us with machine
>>> learning
>>>> tools like submarine.
>>>> 
>>>> I wish hadoop submarine is getting better and better.
>>>> 
>>>> 
>>>> ==============================
>>>> 赵欣
>>>> 东南大学电气工程学院
>>>> 
>>>> -----------------------------------------------------
>>>> 
>>>> Zhao XIN
>>>> 
>>>> School of Electrical Engineering
>>>> 
>>>> ==============================
>>>> 2019-07-18
>>>> 
>>>> 
>>>> *From:* Xun Liu <li...@apache.org>
>>>> *Date:* 2019-07-18 09:46
>>>> *To:* xinzhao <xi...@seu.edu.cn>
>>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>>>> project?
>>>> 
>>>> 
>>>> ---------- Forwarded message ---------
>>>> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
>>>> Date: 2019年7月17日周三 下午3:17
>>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
>>>> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
>>>> runlin512@gmail.com>
>>>> Cc: Xun Liu <li...@apache.org>, common-dev <
>>> common-dev@hadoop.apache.org>,
>>>> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
>>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>>>> submarine-dev@hadoop.apache.org>
>>>> 
>>>> 
>>>> +1 ，Good idea, we are very much looking forward to it.
>>>> 
>>>> ------------------------------
>>>> dashuiguailuyun@gmail.com
>>>> 
>>>> 
>>>> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
>>>> *Date:* 2019-07-17 14:55
>>>> *To:* runlin zhang <ru...@gmail.com>
>>>> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
>>>> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
>>>> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
>>>> <ma...@hadoop.apache.org>; submarine-dev
>>>> <su...@hadoop.apache.org>
>>>> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
>>>> +1, this is a very great idea.
>>>> As Hadoop repository has already grown huge and contains many
>> projects, I
>>>> think in general it's a good idea to separate projects in the early
>>> phase.
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>>>> 
>>>>> +1 ，That will be great ！
>>>>> 
>>>>>> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> This is Xun Liu contributing to the Submarine project for deep
>>> learning
>>>>>> workloads running with big data workloads together on Hadoop
>>> clusters.
>>>>>> 
>>>>>> There are a bunch of integrations of Submarine to other projects
>> are
>>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
>>> next
>>>>> step
>>>>>> of Submarine is going to integrate with more projects like Apache
>>>> Arrow,
>>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine learning
>>> use
>>>>>> cases like model serving, notebook management, advanced training
>>>>>> optimizations (like auto parameter tuning, memory cache
>> optimizations
>>>> for
>>>>>> large datasets for training, etc.), and make it run on other
>>> platforms
>>>>> like
>>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
>>>>> project
>>>>>> to Apache so we can put Submarine and TonY together to the same
>>>> codebase
>>>>>> (Page #30.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>>>>>> ).
>>>>>> 
>>>>>> This expands the scope of the original Submarine project in
>> exciting
>>>> new
>>>>>> ways. Toward that end, would it make sense to create a separate
>>>> Submarine
>>>>>> project at Apache? This can make faster adoption of Submarine, and
>>>> allow
>>>>>> Submarine to grow to a full-blown machine learning platform.
>>>>>> 
>>>>>> There will be lots of technical details to work out, but any
>> initial
>>>>>> thoughts on this?
>>>>>> 
>>>>>> Best Regards,
>>>>>> Xun Liu
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan <wh...@gmail.com>]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo <ne...@pingcap.com> wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==============================
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -----------------------------------------------------
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==============================
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu <li...@apache.org>
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao <xi...@seu.edu.cn>
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > ---------- Forwarded message ---------
> > > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > > runlin512@gmail.com>
> > > Cc: Xun Liu <li...@apache.org>, common-dev <
> > common-dev@hadoop.apache.org>,
> > > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-dev@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > ------------------------------
> > > dashuiguailuyun@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang <ru...@gmail.com>
> > > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > > <ma...@hadoop.apache.org>; submarine-dev
> > > <su...@hadoop.apache.org>
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the original Submarine project in
> exciting
> > > new
> > > > > ways. Toward that end, would it make sense to create a separate
> > > Submarine
> > > > > project at Apache? This can make faster adoption of Submarine, and
> > > allow
> > > > > Submarine to grow to a full-blown machine learning platform.
> > > > >
> > > > > There will be lots of technical details to work out, but any
> initial
> > > > > thoughts on this?
> > > > >
> > > > > Best Regards,
> > > > > Xun Liu
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > > >
> > >
> > >
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[neo <ne...@pingcap.com>]

+1, This is neo from TiDB & TiKV community.
Thanks Xun for bring this up.

Our CNCF project's open source distributed KV storage system TiKV,
Hadoop submarine's machine learning engine helps us to optimize data
storage,
helping us solve some problems in data hotspots and data shuffers.

We are ready to improve the performance of TiDB in our open source
distributed relational database TiDB and also using the hadoop submarine
machine learning engine.

I think if submarine can be independent, it will develop faster and better.
Thanks to the hadoop community for developing submarine!

Best Regards,
neo
www.pingcap.com / https://github.com/pingcap/tidb / https://github.com/tikv

Xun Liu <li...@apache.org> 于2019年7月22日周一 下午4:07写道：

> @adam.antal
>
> The submarine development team has completed the following preparations:
> 1. Established a temporary test repository on Github.
> 2. Change the package name of hadoop submarine from org.hadoop.submarine to
> org.submarine
> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> 4. On the Github docked travis-ci system, all test cases have been tested;
> 5. Several Hadoop submarine users completed the system test using the code
> in this repository.
>
> 赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
>
> > Hi
> >
> > I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> > a major in electrical engineering. Our teaching teams and students use
> > bigoop submarine for big data analysis and automation control of
> electrical
> > equipment.
> >
> > Many thanks to the hadoop community for providing us with machine
> learning
> > tools like submarine.
> >
> > I wish hadoop submarine is getting better and better.
> >
> >
> > ==============================
> > 赵欣
> > 东南大学电气工程学院
> >
> > -----------------------------------------------------
> >
> > Zhao XIN
> >
> > School of Electrical Engineering
> >
> > ==============================
> > 2019-07-18
> >
> >
> > *From:* Xun Liu <li...@apache.org>
> > *Date:* 2019-07-18 09:46
> > *To:* xinzhao <xi...@seu.edu.cn>
> > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > project?
> >
> >
> > ---------- Forwarded message ---------
> > 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> > Date: 2019年7月17日周三 下午3:17
> > Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> > To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> > runlin512@gmail.com>
> > Cc: Xun Liu <li...@apache.org>, common-dev <
> common-dev@hadoop.apache.org>,
> > yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> > hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > submarine-dev@hadoop.apache.org>
> >
> >
> > +1 ，Good idea, we are very much looking forward to it.
> >
> > ------------------------------
> > dashuiguailuyun@gmail.com
> >
> >
> > *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> > *Date:* 2019-07-17 14:55
> > *To:* runlin zhang <ru...@gmail.com>
> > *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> > <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> > Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> > <ma...@hadoop.apache.org>; submarine-dev
> > <su...@hadoop.apache.org>
> > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > +1, this is a very great idea.
> > As Hadoop repository has already grown huge and contains many projects, I
> > think in general it's a good idea to separate projects in the early
> phase.
> >
> >
> > On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
> >
> > > +1 ，That will be great ！
> > >
> > > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[赵欣 <xi...@seu.edu.cn>]

Hi

I am a teacher at Southeast University (https://www.seu.edu.cn/). We are a major in electrical engineering. Our teaching teams and students use bigoop submarine for big data analysis and automation control of electrical equipment.

Many thanks to the hadoop community for providing us with machine learning tools like submarine.

I wish hadoop submarine is getting better and better.
 
==============================
赵欣
东南大学电气工程学院
-----------------------------------------------------
Zhao XIN
School of Electrical Engineering
==============================
2019-07-22
 
From: Xun Liu
Date: 2019-07-22 16:06
To: 赵欣; adam.antal
CC: common-dev; yarn-dev; hdfs-dev; mapreduce-dev; submarine-dev; user
Subject: Re: Any thoughts making Submarine a separate Apache project?
@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：
Hi

I am a teacher at Southeast University (https://www.seu.edu.cn/). We are a major in electrical engineering. Our teaching teams and students use bigoop submarine for big data analysis and automation control of electrical equipment.

Many thanks to the hadoop community for providing us with machine learning tools like submarine.

I wish hadoop submarine is getting better and better.

 
==============================
赵欣
东南大学电气工程学院
-----------------------------------------------------
Zhao XIN
School of Electrical Engineering
==============================
2019-07-18
 
From: Xun Liu
Date: 2019-07-18 09:46
To: xinzhao
Subject: Fwd: Re: Any thoughts making Submarine a separate Apache project?


---------- Forwarded message ---------
发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
Date: 2019年7月17日周三 下午3:17
Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <ru...@gmail.com>
Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>, yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <hd...@hadoop.apache.org>, mapreduce-dev <ma...@hadoop.apache.org>, submarine-dev <su...@hadoop.apache.org>


+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Xun Liu <li...@apache.org>]

@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <xi...@seu.edu.cn> 于2019年7月22日周一 上午9:38写道：

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <li...@apache.org>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <xi...@seu.edu.cn>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Fred <da...@gmail.com>]

Hi

I am a user of hadoop submarine. I have been serving our customers through
hadoop submarine for some time. If hadoop submarine can update the version
a little faster, the algorithm development is carried out by zeppelin,
jupyter, and the model online service is provided through k8s. Let our
users be more satisfied.

Thanks


On Wed, Jul 17, 2019 at 7:07 PM Xun Liu <li...@apache.org> wrote:

>
>
> ---------- Forwarded message ---------
> 发件人： dashuiguailuyun@gmail.com <da...@gmail.com>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <sn...@cloudera.com.invalid>, runlin zhang <
> runlin512@gmail.com>
> Cc: Xun Liu <li...@apache.org>, common-dev <co...@hadoop.apache.org>,
> yarn-dev <ya...@hadoop.apache.org>, hdfs-dev <
> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>, submarine-dev <
> submarine-dev@hadoop.apache.org>
>
>
> +1 ，Good idea, we are very much looking forward to it.
>
> ------------------------------
> dashuiguailuyun@gmail.com
>
>
> *From:* Szilard Nemeth <sn...@cloudera.com.INVALID>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <ru...@gmail.com>
> *CC:* Xun Liu <li...@apache.org>; Hadoop Common
> <co...@hadoop.apache.org>; yarn-dev <ya...@hadoop.apache.org>;
> Hdfs-dev <hd...@hadoop.apache.org>; mapreduce-dev
> <ma...@hadoop.apache.org>; submarine-dev
> <su...@hadoop.apache.org>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
>
> > +1 ，That will be great ！
> >
> > > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1 ，Good idea, we are very much looking forward to it.



dashuiguailuyun@gmail.com
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:
 
> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Szilard Nemeth <sn...@cloudera.com.INVALID>]

+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.


On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:

> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Szilard Nemeth <sn...@cloudera.com.INVALID>]

+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.


On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:

> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Szilard Nemeth <sn...@cloudera.com.INVALID>]

+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.


On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:

> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Szilard Nemeth <sn...@cloudera.com.INVALID>]

+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.


On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:

> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Any thoughts making Submarine a separate Apache project?

[Szilard Nemeth <sn...@cloudera.com.INVALID>]

+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.


On Wed, Jul 17, 2019, 08:50 runlin zhang <ru...@gmail.com> wrote:

> +1 ，That will be great ！
>
> > 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Fwd: Any thoughts making Submarine a separate Apache project?

[蒋晓峰 <th...@gmail.com>]

Hi,

I am a user of hadoop submarine. I have used hadoop submarine for more than
half a year at our company https://www.lu.com/, mainly for offline learning
of machine learning. Hadoop submarine can run well in our yarn. Above the
cluster, we don't need to learn other techniques to complete the machine
learning work. Hadoop submarine is very helpful to us. If the hadoop
submarine can develop independently, if it can be combined with jupyter and
k8s, I think it will develop better.


+1

---------- Forwarded message ---------
发件人： runlin zhang <ru...@gmail.com>
Date: 2019年7月17日周三 下午2:50
Subject: Re: Any thoughts making Submarine a separate Apache project?
To: Xun Liu <li...@apache.org>
Cc: Hadoop Common <co...@hadoop.apache.org>, <
yarn-dev@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, <
mapreduce-dev@hadoop.apache.org>, <su...@hadoop.apache.org>


+1 ，That will be great ！

> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>
> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms
like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu

Fwd: Any thoughts making Submarine a separate Apache project?

[蒋晓峰 <th...@gmail.com>]

Hi,

I am a user of hadoop submarine. I have used hadoop submarine for more than
half a year at our company https://www.lu.com/, mainly for offline learning
of machine learning. Hadoop submarine can run well in our yarn. Above the
cluster, we don't need to learn other techniques to complete the machine
learning work. Hadoop submarine is very helpful to us. If the hadoop
submarine can develop independently, if it can be combined with jupyter and
k8s, I think it will develop better.


+1

---------- Forwarded message ---------
发件人： runlin zhang <ru...@gmail.com>
Date: 2019年7月17日周三 下午2:50
Subject: Re: Any thoughts making Submarine a separate Apache project?
To: Xun Liu <li...@apache.org>
Cc: Hadoop Common <co...@hadoop.apache.org>, <
yarn-dev@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, <
mapreduce-dev@hadoop.apache.org>, <su...@hadoop.apache.org>


+1 ，That will be great ！

> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
>
> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms
like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu

Re: Any thoughts making Submarine a separate Apache project?

[runlin zhang <ru...@gmail.com>]

+1 ，That will be great ！

> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[zac yuan <yu...@gmail.com>]

That is a great idea. Looking forward to combining submarine and tony
together.

+1 (non-binding)

Zac Zhou

Xun Liu <li...@apache.org> 于2019年7月10日周三 下午3:35写道：

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji <wa...@gmail.com>]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu <li...@apache.org> wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji <wa...@gmail.com>]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu <li...@apache.org> wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Adam Antal <ad...@cloudera.com.INVALID>]

+1 (non-binding). Good initiative!

A question to someone who has more insight on this area: how much effort
would that mean - besides the straightforward maven work (modifying the
pom.xmls).

On Fri, Jul 19, 2019 at 10:40 AM dashuiguailuyun@gmail.com <
dashuiguailuyun@gmail.com> wrote:

> +1.Submarine is already in use at our company（贝壳找房） and is performing.
> Looking forward to the next step to provide more features
>
>
>
> dashuiguailuyun@gmail.com
>
> From: Oliver Hu
> Date: 2019-07-19 07:50
> To: Jeff Zhang
> CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev;
> submarine-dev
> Subject: Re: Any thoughts making Submarine a separate Apache project?
> +1 (non-binding). Make Submarine a separate project would make it easier to
> integrate with other components in the ML pipeline and expand cross
> platform.
>
> On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
>
> > +1， This is Jeff Zhang from Zeppelin community.
> > Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> > several months ago, and I already see some early adoption of that in
> China.
> > AI is fast growing area, I believe moving into a separate project would
> be
> > helpful for Submarine to catch up with the new trend of AI and release
> more
> > new features quickly than before.
> >
> >
> >
> > sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
> >
> > > +1  We are look forward to it. The idea is great.
> > >
> > > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Adam Antal <ad...@cloudera.com.INVALID>]

+1 (non-binding). Good initiative!

A question to someone who has more insight on this area: how much effort
would that mean - besides the straightforward maven work (modifying the
pom.xmls).

On Fri, Jul 19, 2019 at 10:40 AM dashuiguailuyun@gmail.com <
dashuiguailuyun@gmail.com> wrote:

> +1.Submarine is already in use at our company（贝壳找房） and is performing.
> Looking forward to the next step to provide more features
>
>
>
> dashuiguailuyun@gmail.com
>
> From: Oliver Hu
> Date: 2019-07-19 07:50
> To: Jeff Zhang
> CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev;
> submarine-dev
> Subject: Re: Any thoughts making Submarine a separate Apache project?
> +1 (non-binding). Make Submarine a separate project would make it easier to
> integrate with other components in the ML pipeline and expand cross
> platform.
>
> On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
>
> > +1， This is Jeff Zhang from Zeppelin community.
> > Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> > several months ago, and I already see some early adoption of that in
> China.
> > AI is fast growing area, I believe moving into a separate project would
> be
> > helpful for Submarine to catch up with the new trend of AI and release
> more
> > new features quickly than before.
> >
> >
> >
> > sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
> >
> > > +1  We are look forward to it. The idea is great.
> > >
> > > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Adam Antal <ad...@cloudera.com.INVALID>]

+1 (non-binding). Good initiative!

A question to someone who has more insight on this area: how much effort
would that mean - besides the straightforward maven work (modifying the
pom.xmls).

On Fri, Jul 19, 2019 at 10:40 AM dashuiguailuyun@gmail.com <
dashuiguailuyun@gmail.com> wrote:

> +1.Submarine is already in use at our company（贝壳找房） and is performing.
> Looking forward to the next step to provide more features
>
>
>
> dashuiguailuyun@gmail.com
>
> From: Oliver Hu
> Date: 2019-07-19 07:50
> To: Jeff Zhang
> CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev;
> submarine-dev
> Subject: Re: Any thoughts making Submarine a separate Apache project?
> +1 (non-binding). Make Submarine a separate project would make it easier to
> integrate with other components in the ML pipeline and expand cross
> platform.
>
> On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
>
> > +1， This is Jeff Zhang from Zeppelin community.
> > Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> > several months ago, and I already see some early adoption of that in
> China.
> > AI is fast growing area, I believe moving into a separate project would
> be
> > helpful for Submarine to catch up with the new trend of AI and release
> more
> > new features quickly than before.
> >
> >
> >
> > sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
> >
> > > +1  We are look forward to it. The idea is great.
> > >
> > > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Adam Antal <ad...@cloudera.com.INVALID>]

+1 (non-binding). Good initiative!

A question to someone who has more insight on this area: how much effort
would that mean - besides the straightforward maven work (modifying the
pom.xmls).

On Fri, Jul 19, 2019 at 10:40 AM dashuiguailuyun@gmail.com <
dashuiguailuyun@gmail.com> wrote:

> +1.Submarine is already in use at our company（贝壳找房） and is performing.
> Looking forward to the next step to provide more features
>
>
>
> dashuiguailuyun@gmail.com
>
> From: Oliver Hu
> Date: 2019-07-19 07:50
> To: Jeff Zhang
> CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev;
> submarine-dev
> Subject: Re: Any thoughts making Submarine a separate Apache project?
> +1 (non-binding). Make Submarine a separate project would make it easier to
> integrate with other components in the ML pipeline and expand cross
> platform.
>
> On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
>
> > +1， This is Jeff Zhang from Zeppelin community.
> > Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> > several months ago, and I already see some early adoption of that in
> China.
> > AI is fast growing area, I believe moving into a separate project would
> be
> > helpful for Submarine to catch up with the new trend of AI and release
> more
> > new features quickly than before.
> >
> >
> >
> > sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
> >
> > > +1  We are look forward to it. The idea is great.
> > >
> > > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

[Adam Antal <ad...@cloudera.com.INVALID>]

+1 (non-binding). Good initiative!

A question to someone who has more insight on this area: how much effort
would that mean - besides the straightforward maven work (modifying the
pom.xmls).

On Fri, Jul 19, 2019 at 10:40 AM dashuiguailuyun@gmail.com <
dashuiguailuyun@gmail.com> wrote:

> +1.Submarine is already in use at our company（贝壳找房） and is performing.
> Looking forward to the next step to provide more features
>
>
>
> dashuiguailuyun@gmail.com
>
> From: Oliver Hu
> Date: 2019-07-19 07:50
> To: Jeff Zhang
> CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev;
> submarine-dev
> Subject: Re: Any thoughts making Submarine a separate Apache project?
> +1 (non-binding). Make Submarine a separate project would make it easier to
> integrate with other components in the ML pipeline and expand cross
> platform.
>
> On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
>
> > +1， This is Jeff Zhang from Zeppelin community.
> > Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> > several months ago, and I already see some early adoption of that in
> China.
> > AI is fast growing area, I believe moving into a separate project would
> be
> > helpful for Submarine to catch up with the new trend of AI and release
> more
> > new features quickly than before.
> >
> >
> >
> > sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
> >
> > > +1  We are look forward to it. The idea is great.
> > >
> > > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is Xun Liu contributing to the Submarine project for deep
> learning
> > > > workloads running with big data workloads together on Hadoop
> clusters.
> > > >
> > > > There are a bunch of integrations of Submarine to other projects are
> > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> next
> > > step
> > > > of Submarine is going to integrate with more projects like Apache
> > Arrow,
> > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> use
> > > > cases like model serving, notebook management, advanced training
> > > > optimizations (like auto parameter tuning, memory cache optimizations
> > for
> > > > large datasets for training, etc.), and make it run on other
> platforms
> > > like
> > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > project
> > > > to Apache so we can put Submarine and TonY together to the same
> > codebase
> > > > (Page #30.
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > ).
> > > >
> > > > This expands the scope of the original Submarine project in exciting
> > new
> > > > ways. Toward that end, would it make sense to create a separate
> > Submarine
> > > > project at Apache? This can make faster adoption of Submarine, and
> > allow
> > > > Submarine to grow to a full-blown machine learning platform.
> > > >
> > > > There will be lots of technical details to work out, but any initial
> > > > thoughts on this?
> > > >
> > > > Best Regards,
> > > > Xun Liu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1.Submarine is already in use at our company（贝壳找房） and is performing. Looking forward to the next step to provide more features



dashuiguailuyun@gmail.com
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
 
> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1.Submarine is already in use at our company（贝壳找房） and is performing. Looking forward to the next step to provide more features



dashuiguailuyun@gmail.com
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
 
> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1.Submarine is already in use at our company（贝壳找房） and is performing. Looking forward to the next step to provide more features



dashuiguailuyun@gmail.com
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
 
> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1.Submarine is already in use at our company（贝壳找房） and is performing. Looking forward to the next step to provide more features



dashuiguailuyun@gmail.com
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
 
> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Re: Any thoughts making Submarine a separate Apache project?

["dashuiguailuyun@gmail.com" <da...@gmail.com>]

+1.Submarine is already in use at our company（贝壳找房） and is performing. Looking forward to the next step to provide more features



dashuiguailuyun@gmail.com
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:
 
> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Any thoughts making Submarine a separate Apache project?

[Oliver Hu <ol...@gmail.com>]

+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.

On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:

> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Any thoughts making Submarine a separate Apache project?

[Oliver Hu <ol...@gmail.com>]

+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.

On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <zj...@gmail.com> wrote:

> +1， This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Any thoughts making Submarine a separate Apache project?

[Jeff Zhang <zj...@gmail.com>]

+1， This is Jeff Zhang from Zeppelin community.
Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
several months ago, and I already see some early adoption of that in China.
AI is fast growing area, I believe moving into a separate project would be
helpful for Submarine to catch up with the new trend of AI and release more
new features quickly than before.



sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：

> +1  We are look forward to it. The idea is great.
>
> > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

-- 
Best Regards

Jeff Zhang

Re: Any thoughts making Submarine a separate Apache project?

[Jeff Zhang <zj...@gmail.com>]

+1， This is Jeff Zhang from Zeppelin community.
Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
several months ago, and I already see some early adoption of that in China.
AI is fast growing area, I believe moving into a separate project would be
helpful for Submarine to catch up with the new trend of AI and release more
new features quickly than before.



sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：

> +1  We are look forward to it. The idea is great.
>
> > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

-- 
Best Regards

Jeff Zhang

Re: Any thoughts making Submarine a separate Apache project?

[Jeff Zhang <zj...@gmail.com>]

+1， This is Jeff Zhang from Zeppelin community.
Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
several months ago, and I already see some early adoption of that in China.
AI is fast growing area, I believe moving into a separate project would be
helpful for Submarine to catch up with the new trend of AI and release more
new features quickly than before.



sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：

> +1  We are look forward to it. The idea is great.
>
> > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

-- 
Best Regards

Jeff Zhang

Re: Any thoughts making Submarine a separate Apache project?

[Jeff Zhang <zj...@gmail.com>]

+1， This is Jeff Zhang from Zeppelin community.
Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
several months ago, and I already see some early adoption of that in China.
AI is fast growing area, I believe moving into a separate project would be
helpful for Submarine to catch up with the new trend of AI and release more
new features quickly than before.



sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：

> +1  We are look forward to it. The idea is great.
>
> > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

-- 
Best Regards

Jeff Zhang

Re: Any thoughts making Submarine a separate Apache project?

[Jeff Zhang <zj...@gmail.com>]

+1， This is Jeff Zhang from Zeppelin community.
Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
several months ago, and I already see some early adoption of that in China.
AI is fast growing area, I believe moving into a separate project would be
helpful for Submarine to catch up with the new trend of AI and release more
new features quickly than before.



sid yu <ji...@gmail.com> 于2019年7月18日周四 下午2:06写道：

> +1  We are look forward to it. The idea is great.
>
> > On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

-- 
Best Regards

Jeff Zhang

Re: Any thoughts making Submarine a separate Apache project?

[sid yu <ji...@gmail.com>]

+1  We are look forward to it. The idea is great.

> On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu

Re: Any thoughts making Submarine a separate Apache project?

[runlin zhang <ru...@gmail.com>]

+1 ，That will be great ！

> 在 2019年7月10日，下午3:34，Xun Liu <li...@apache.org> 写道：
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji <wa...@gmail.com>]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu <li...@apache.org> wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Any thoughts making Submarine a separate Apache project?

[zac yuan <yu...@gmail.com>]

That is a great idea. Looking forward to combining submarine and tony
together.

+1 (non-binding)

Zac Zhou

Xun Liu <li...@apache.org> 于2019年7月10日周三 下午3:35写道：

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji <wa...@gmail.com>]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu <li...@apache.org> wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

Re: Any thoughts making Submarine a separate Apache project?

[sid yu <ji...@gmail.com>]

+1  We are look forward to it. The idea is great.

> On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[sid yu <ji...@gmail.com>]

+1  We are look forward to it. The idea is great.

> On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu

Re: Any thoughts making Submarine a separate Apache project?

[sid yu <ji...@gmail.com>]

+1  We are look forward to it. The idea is great.

> On Jul 10, 2019, at 3:34 PM, Xun Liu <li...@apache.org> wrote:
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji <wa...@gmail.com>]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu <li...@apache.org> wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>