Re: Zeppelin add hadoop submarine(machine learning) interpreter

[liuxun <ne...@163.com>]

hi, community

I updated the design documentation, 
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
added the interactive development of machine learning algorithms and the overall design of submitting Note packages to YARN for job execution.

I have completed the development of the pre-research part of the code, and established JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-3856 <https://issues.apache.org/jira/browse/ZEPPELIN-3856>

welcome comments, thank you!

> 在 2018年10月30日，下午10:11，liuxun <ne...@163.com> 写道：
> 
> HI, community
> 
> 
> I updated the design document and submitted it to the hadoop community.
> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
> 
> If you have any questions, you can ask them in the email or directly in the document. Thank you!
> 
> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
> 
> 1. zeppelin submarine interpreter properties
> <submarine-interpreter.png>
> 1.1 System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
> 1.2 Allow different users to use different interpreters for resource allocation and management.
> 
> 2. zeppelin submarine Tensorflow interpreter
> <submarine-tensorflow.png>
> 2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
> 2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
> 
> 3. zeppelin submarine interpreter
> <submarine-command.png>
> 3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
> 3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
> 3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
> 3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
> 
> 
>> 在 2018年10月30日，下午6:40，liuxun <neliuxun@163.com <ma...@163.com>> 写道：
>> 
>> 
>> HI,
>> 
>> 
>> 	I updated the design document and submitted it to the hadoop community.
>> 	https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>> 
>> 	If you have any questions, you can ask them in the email or directly in the document. Thank you!
>> 
>> 	I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>> 
>> zeppelin submarine interpreter properties
>> 
>> System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
>> Allow different users to use different interpreters for resource allocation and management.
>> zeppelin submarine Tensorflow interpreter
>> 
>> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
>> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>> 
>> zeppelin submarine interpreter
>> 
>> The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
>> The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
>> The progress and log information of the submarine will be displayed in zeppelin's note.
>> You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>> 
>> 
>> 
>>> 在 2018年10月21日，上午6:26，Felix Cheung <felixcheung_m@hotmail.com <ma...@hotmail.com>> 写道：
>>> 
>>> Very cool!
>>> 
>>> 
>>> ________________________________
>>> From: Jeff Zhang <zjffdu@gmail.com <ma...@gmail.com>>
>>> Sent: Friday, October 19, 2018 7:14 AM
>>> To: dev@zeppelin.apache.org <ma...@zeppelin.apache.org>
>>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>>> 
>>> Thanks xun. This would be a great addon for zeppelin to support deep
>>> learning. I will check the design later.
>>> 
>>> 
>>> 
>>> liu xun <neliuxun@163.com <ma...@163.com>>于2018年10月19日周五 下午3:56写道：
>>> 
>>>> Hi,
>>>> Hadoop Submarine is the latest machine learning framework subproject in
>>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>>> full-featured system framework for machine learning algorithm development,
>>>> distributed model training, model management, and model publishing,
>>>> combined with hadoop's intrinsic data storage and data processing
>>>> capabilities to enable data scientists to Good mining and the value of the
>>>> data.
>>>> 
>>>> 
>>>> I was involved in the development of the hadoop submarine project. So I
>>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>>> Let zeppeline increase the development of deep learning. This is my design
>>>> document, let's see if there is any opinion, you can put it directly in the
>>>> document, thank you!
>>>> 
>>>> 
>>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd>
>>>> 
>>>> 
>>>> 
>> 
>