You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by liuxun <ne...@163.com> on 2018/10/30 14:11:34 UTC
Re: Zeppelin add hadoop submarine(machine learning framework)
interpreter
HI, community
I updated the design document and submitted it to the hadoop community.
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
If you have any questions, you can ask them in the email or directly in the document. Thank you!
I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
1. zeppelin submarine interpreter properties
1.1 System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
1.2 Allow different users to use different interpreters for resource allocation and management.
2. zeppelin submarine Tensorflow interpreter
2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
3. zeppelin submarine interpreter
3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
> 在 2018年10月30日，下午6:40，liuxun <ne...@163.com> 写道：
>
>
> HI,
>
>
> I updated the design document and submitted it to the hadoop community.
> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>
> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>
> zeppelin submarine interpreter properties
>
> System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
> Allow different users to use different interpreters for resource allocation and management.
> zeppelin submarine Tensorflow interpreter
>
> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>
> zeppelin submarine interpreter
>
> The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
> The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
> The progress and log information of the submarine will be displayed in zeppelin's note.
> You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>
>
>
>> 在 2018年10月21日，上午6:26，Felix Cheung <fe...@hotmail.com> 写道：
>>
>> Very cool!
>>
>>
>> ________________________________
>> From: Jeff Zhang <zj...@gmail.com>
>> Sent: Friday, October 19, 2018 7:14 AM
>> To: dev@zeppelin.apache.org
>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>>
>> Thanks xun. This would be a great addon for zeppelin to support deep
>> learning. I will check the design later.
>>
>>
>>
>> liu xun <ne...@163.com>于2018年10月19日周五 下午3:56写道：
>>
>>> Hi,
>>> Hadoop Submarine is the latest machine learning framework subproject in
>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>> full-featured system framework for machine learning algorithm development,
>>> distributed model training, model management, and model publishing,
>>> combined with hadoop's intrinsic data storage and data processing
>>> capabilities to enable data scientists to Good mining and the value of the
>>> data.
>>>
>>>
>>> I was involved in the development of the hadoop submarine project. So I
>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>> Let zeppeline increase the development of deep learning. This is my design
>>> document, let's see if there is any opinion, you can put it directly in the
>>> document, thank you!
>>>
>>>
>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd
>>>
>>>
>>>
>
Re: Zeppelin add hadoop submarine(machine learning) interpreter
Posted by liuxun <ne...@163.com>.
hi, community
I updated the design documentation,
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
added the interactive development of machine learning algorithms and the overall design of submitting Note packages to YARN for job execution.
I have completed the development of the pre-research part of the code, and established JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-3856 <https://issues.apache.org/jira/browse/ZEPPELIN-3856>
welcome comments, thank you!
> 在 2018年10月30日，下午10:11，liuxun <ne...@163.com> 写道：
>
> HI, community
>
>
> I updated the design document and submitted it to the hadoop community.
> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
>
> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>
> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>
> 1. zeppelin submarine interpreter properties
> <submarine-interpreter.png>
> 1.1 System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
> 1.2 Allow different users to use different interpreters for resource allocation and management.
>
> 2. zeppelin submarine Tensorflow interpreter
> <submarine-tensorflow.png>
> 2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
> 2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>
> 3. zeppelin submarine interpreter
> <submarine-command.png>
> 3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
> 3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
> 3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
> 3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>
>
>> 在 2018年10月30日，下午6:40，liuxun <neliuxun@163.com <ma...@163.com>> 写道：
>>
>>
>> HI,
>>
>>
>> I updated the design document and submitted it to the hadoop community.
>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>>
>> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>>
>> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>>
>> zeppelin submarine interpreter properties
>>
>> System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
>> Allow different users to use different interpreters for resource allocation and management.
>> zeppelin submarine Tensorflow interpreter
>>
>> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
>> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>>
>> zeppelin submarine interpreter
>>
>> The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
>> The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
>> The progress and log information of the submarine will be displayed in zeppelin's note.
>> You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>>
>>
>>
>>> 在 2018年10月21日，上午6:26，Felix Cheung <felixcheung_m@hotmail.com <ma...@hotmail.com>> 写道：
>>>
>>> Very cool!
>>>
>>>
>>> ________________________________
>>> From: Jeff Zhang <zjffdu@gmail.com <ma...@gmail.com>>
>>> Sent: Friday, October 19, 2018 7:14 AM
>>> To: dev@zeppelin.apache.org <ma...@zeppelin.apache.org>
>>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>>>
>>> Thanks xun. This would be a great addon for zeppelin to support deep
>>> learning. I will check the design later.
>>>
>>>
>>>
>>> liu xun <neliuxun@163.com <ma...@163.com>>于2018年10月19日周五 下午3:56写道：
>>>
>>>> Hi,
>>>> Hadoop Submarine is the latest machine learning framework subproject in
>>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>>> full-featured system framework for machine learning algorithm development,
>>>> distributed model training, model management, and model publishing,
>>>> combined with hadoop's intrinsic data storage and data processing
>>>> capabilities to enable data scientists to Good mining and the value of the
>>>> data.
>>>>
>>>>
>>>> I was involved in the development of the hadoop submarine project. So I
>>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>>> Let zeppeline increase the development of deep learning. This is my design
>>>> document, let's see if there is any opinion, you can put it directly in the
>>>> document, thank you!
>>>>
>>>>
>>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd>
>>>>
>>>>
>>>>
>>
>
Re: Zeppelin add hadoop submarine(machine learning) interpreter
Posted by liuxun <ne...@163.com>.
hi, community
I updated the design documentation,
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
added the interactive development of machine learning algorithms and the overall design of submitting Note packages to YARN for job execution.
I have completed the development of the pre-research part of the code, and established JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-3856 <https://issues.apache.org/jira/browse/ZEPPELIN-3856>
welcome comments, thank you!
> 在 2018年10月30日，下午10:11，liuxun <ne...@163.com> 写道：
>
> HI, community
>
>
> I updated the design document and submitted it to the hadoop community.
> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
>
> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>
> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>
> 1. zeppelin submarine interpreter properties
> <submarine-interpreter.png>
> 1.1 System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
> 1.2 Allow different users to use different interpreters for resource allocation and management.
>
> 2. zeppelin submarine Tensorflow interpreter
> <submarine-tensorflow.png>
> 2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
> 2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>
> 3. zeppelin submarine interpreter
> <submarine-command.png>
> 3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
> 3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
> 3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
> 3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>
>
>> 在 2018年10月30日，下午6:40，liuxun <neliuxun@163.com <ma...@163.com>> 写道：
>>
>>
>> HI,
>>
>>
>> I updated the design document and submitted it to the hadoop community.
>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>>
>> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>>
>> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>>
>> zeppelin submarine interpreter properties
>>
>> System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
>> Allow different users to use different interpreters for resource allocation and management.
>> zeppelin submarine Tensorflow interpreter
>>
>> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
>> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>>
>> zeppelin submarine interpreter
>>
>> The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
>> The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
>> The progress and log information of the submarine will be displayed in zeppelin's note.
>> You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>>
>>
>>
>>> 在 2018年10月21日，上午6:26，Felix Cheung <felixcheung_m@hotmail.com <ma...@hotmail.com>> 写道：
>>>
>>> Very cool!
>>>
>>>
>>> ________________________________
>>> From: Jeff Zhang <zjffdu@gmail.com <ma...@gmail.com>>
>>> Sent: Friday, October 19, 2018 7:14 AM
>>> To: dev@zeppelin.apache.org <ma...@zeppelin.apache.org>
>>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>>>
>>> Thanks xun. This would be a great addon for zeppelin to support deep
>>> learning. I will check the design later.
>>>
>>>
>>>
>>> liu xun <neliuxun@163.com <ma...@163.com>>于2018年10月19日周五 下午3:56写道：
>>>
>>>> Hi,
>>>> Hadoop Submarine is the latest machine learning framework subproject in
>>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>>> full-featured system framework for machine learning algorithm development,
>>>> distributed model training, model management, and model publishing,
>>>> combined with hadoop's intrinsic data storage and data processing
>>>> capabilities to enable data scientists to Good mining and the value of the
>>>> data.
>>>>
>>>>
>>>> I was involved in the development of the hadoop submarine project. So I
>>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>>> Let zeppeline increase the development of deep learning. This is my design
>>>> document, let's see if there is any opinion, you can put it directly in the
>>>> document, thank you!
>>>>
>>>>
>>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd>
>>>>
>>>>
>>>>
>>
>
Re: Zeppelin add hadoop submarine(machine learning) interpreter
Posted by liuxun <ne...@163.com>.
hi, community
I updated the design documentation,
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
added the interactive development of machine learning algorithms and the overall design of submitting Note packages to YARN for job execution.
I have completed the development of the pre-research part of the code, and established JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-3856 <https://issues.apache.org/jira/browse/ZEPPELIN-3856>
welcome comments, thank you!
> 在 2018年10月30日，下午10:11，liuxun <ne...@163.com> 写道：
>
> HI, community
>
>
> I updated the design document and submitted it to the hadoop community.
> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>
>
> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>
> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>
> 1. zeppelin submarine interpreter properties
> <submarine-interpreter.png>
> 1.1 System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
> 1.2 Allow different users to use different interpreters for resource allocation and management.
>
> 2. zeppelin submarine Tensorflow interpreter
> <submarine-tensorflow.png>
> 2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
> 2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>
> 3. zeppelin submarine interpreter
> <submarine-command.png>
> 3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
> 3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
> 3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
> 3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>
>
>> 在 2018年10月30日，下午6:40，liuxun <neliuxun@163.com <ma...@163.com>> 写道：
>>
>>
>> HI,
>>
>>
>> I updated the design document and submitted it to the hadoop community.
>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>>
>> If you have any questions, you can ask them in the email or directly in the document. Thank you!
>>
>> I developed the submarine interpreter module in zeppelin. Below is the interface for the system to run.
>>
>> zeppelin submarine interpreter properties
>>
>> System administrators can configure multiple interpreters, and different interpreters can have different resource configurations.
>> Allow different users to use different interpreters for resource allocation and management.
>> zeppelin submarine Tensorflow interpreter
>>
>> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To write python code for tensorflow.
>> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin will upload the python code to the specified HDFS directory. The submarine is loaded into the docker container when it is running.
>>
>> zeppelin submarine interpreter
>>
>> The user sets the call parameter value of the tensorflow python code, Then enter the job run command.
>> The zeppelin submarine interpreter first checks that all parameters of the submarine run are set completely. After the check is passed, the implementation of YARN is performed via submarine.jar.
>> The progress and log information of the submarine will be displayed in zeppelin's note.
>> You can enter submarine's other commands in the submarine interpreter, view all the jobs in the submarine, etc.
>>
>>
>>
>>> 在 2018年10月21日，上午6:26，Felix Cheung <felixcheung_m@hotmail.com <ma...@hotmail.com>> 写道：
>>>
>>> Very cool!
>>>
>>>
>>> ________________________________
>>> From: Jeff Zhang <zjffdu@gmail.com <ma...@gmail.com>>
>>> Sent: Friday, October 19, 2018 7:14 AM
>>> To: dev@zeppelin.apache.org <ma...@zeppelin.apache.org>
>>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>>>
>>> Thanks xun. This would be a great addon for zeppelin to support deep
>>> learning. I will check the design later.
>>>
>>>
>>>
>>> liu xun <neliuxun@163.com <ma...@163.com>>于2018年10月19日周五 下午3:56写道：
>>>
>>>> Hi,
>>>> Hadoop Submarine is the latest machine learning framework subproject in
>>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>>> full-featured system framework for machine learning algorithm development,
>>>> distributed model training, model management, and model publishing,
>>>> combined with hadoop's intrinsic data storage and data processing
>>>> capabilities to enable data scientists to Good mining and the value of the
>>>> data.
>>>>
>>>>
>>>> I was involved in the development of the hadoop submarine project. So I
>>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>>> Let zeppeline increase the development of deep learning. This is my design
>>>> document, let's see if there is any opinion, you can put it directly in the
>>>> document, thank you!
>>>>
>>>>
>>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd>
>>>>
>>>>
>>>>
>>
>