You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Ruslan Dautkhanov <da...@gmail.com> on 2017/01/12 06:32:04 UTC
Passing variables from %pyspark to %sh
It's possible to exchange variables between Scala and Spark
through z.put and z.get.
How to pass a variable to %sh?
In Jupyter it's possible to do for example as
> ! hadoop fs -put {localfile} {hdfsfile}
where localfile and and hdfsfile are Python variables.
Can't find any references for something similar in Shell Interpreter
https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html
In many notebooks we have to pass small variabels
from Zeppelin notes to external scripts as parameters.
It would be awesome to have something like
%sh
> /path/to/script --param8={var1} --param9={var2}
where var1 and var2 would be implied to be fetched as z.get('var1')
and z.get('var2') respectively.
Other thoughts?
Thank you,
Ruslan Dautkhanov
Re: Passing variables from %pyspark to %sh
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Created https://issues.apache.org/jira/browse/ZEPPELIN-1967
(JIRA had some issues.. https://twitter.com/infrabot - had to wait a
couple of days.)
Great ideas. Thank you everyone.
--
Ruslan Dautkhanov
On Thu, Jan 12, 2017 at 8:55 AM, t p <ta...@gmail.com> wrote:
> Is something like feasible from the front end perspective - i.e the web UI
> (Angular?) - i.e. not matter which process/JVM runs the interpreter, I’d
> assume that a book is executed in the context of a we browser which unifies
> all the pages of the book...
>
> On Jan 12, 2017, at 9:56 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Agree to share variables between interpreters. Currently zeppelin launch
> one JVM for each interpreter group. So it is not possible to share
> variables between spark and sh. But for some interpreters like sh, md, it
> is not necessary to create separate JVM for them. We can embed them in
> spark interpreter JVM. But we could not do it for all interpreters,
> because it would cause potential jar conflicts.
>
>
>
> Jongyoul Lee <jo...@gmail.com>于2017年1月12日周四 下午10:18写道：
>
>> Yes, many users suggest that feature to share results between paragraphs
>> and different interpreters. I think this would be one of major features in
>> a next release.
>>
>> On Thu, Jan 12, 2017 at 10:30 PM, t p <ta...@gmail.com> wrote:
>>
>> Is it possible to have similar support to exchange checkbox/dropdown
>> variables and can variables be exchanged with other interpreters like PSQL
>> (e.g. variable set by spark/pyspark and accessible in another para which is
>> running PSQL interpreter).
>>
>> I’m interested in doing this and I’d like to know if there is a way to
>> accomplish this:
>> https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f
>> 90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E
>>
>>
>> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>
>> There's no way to communicate between spark and sh intepreter. It need to
>> implement it but it doesn't yet. But I agree that it would be helpful for
>> some cases. Can you create issue?
>>
>> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <da...@gmail.com>
>> wrote:
>>
>> It's possible to exchange variables between Scala and Spark
>> through z.put and z.get.
>>
>> How to pass a variable to %sh?
>>
>> In Jupyter it's possible to do for example as
>>
>> ! hadoop fs -put {localfile} {hdfsfile}
>>
>>
>> where localfile and and hdfsfile are Python variables.
>>
>> Can't find any references for something similar in Shell Interpreter
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html
>>
>> In many notebooks we have to pass small variabels
>> from Zeppelin notes to external scripts as parameters.
>>
>> It would be awesome to have something like
>>
>> %sh
>> /path/to/script --param8={var1} --param9={var2}
>>
>>
>> where var1 and var2 would be implied to be fetched as z.get('var1')
>> and z.get('var2') respectively.
>>
>> Other thoughts?
>>
>>
>> Thank you,
>> Ruslan Dautkhanov
>>
>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>>
>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
Re: Passing variables from %pyspark to %sh
Posted by t p <ta...@gmail.com>.
Is something like feasible from the front end perspective - i.e the web UI (Angular?) - i.e. not matter which process/JVM runs the interpreter, I’d assume that a book is executed in the context of a we browser which unifies all the pages of the book...
> On Jan 12, 2017, at 9:56 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Agree to share variables between interpreters. Currently zeppelin launch one JVM for each interpreter group. So it is not possible to share variables between spark and sh. But for some interpreters like sh, md, it is not necessary to create separate JVM for them. We can embed them in spark interpreter JVM. But we could not do it for all interpreters, because it would cause potential jar conflicts.
>
>
>
> Jongyoul Lee <jongyoul@gmail.com <ma...@gmail.com>>于2017年1月12日周四 下午10:18写道：
> Yes, many users suggest that feature to share results between paragraphs and different interpreters. I think this would be one of major features in a next release.
>
> On Thu, Jan 12, 2017 at 10:30 PM, t p <tauis2xpi@gmail.com <ma...@gmail.com>> wrote:
> Is it possible to have similar support to exchange checkbox/dropdown variables and can variables be exchanged with other interpreters like PSQL (e.g. variable set by spark/pyspark and accessible in another para which is running PSQL interpreter).
>
> I’m interested in doing this and I’d like to know if there is a way to accomplish this:
> https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E <https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E>
>
>
>> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jongyoul@gmail.com <ma...@gmail.com>> wrote:
>>
>> There's no way to communicate between spark and sh intepreter. It need to implement it but it doesn't yet. But I agree that it would be helpful for some cases. Can you create issue?
>>
>> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <dautkhanov@gmail.com <ma...@gmail.com>> wrote:
>> It's possible to exchange variables between Scala and Spark
>> through z.put and z.get.
>>
>> How to pass a variable to %sh?
>>
>> In Jupyter it's possible to do for example as
>> ! hadoop fs -put {localfile} {hdfsfile}
>>
>> where localfile and and hdfsfile are Python variables.
>>
>> Can't find any references for something similar in Shell Interpreter
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html <https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html>
>>
>> In many notebooks we have to pass small variabels
>> from Zeppelin notes to external scripts as parameters.
>>
>> It would be awesome to have something like
>>
>> %sh
>> /path/to/script --param8={var1} --param9={var2}
>>
>> where var1 and var2 would be implied to be fetched as z.get('var1')
>> and z.get('var2') respectively.
>>
>> Other thoughts?
>>
>>
>> Thank you,
>> Ruslan Dautkhanov
>>
>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net <http://madeng.net/>
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>
Re: Passing variables from %pyspark to %sh
Posted by Jeff Zhang <zj...@gmail.com>.
Agree to share variables between interpreters. Currently zeppelin launch
one JVM for each interpreter group. So it is not possible to share
variables between spark and sh. But for some interpreters like sh, md, it
is not necessary to create separate JVM for them. We can embed them in
spark interpreter JVM. But we could not do it for all interpreters,
because it would cause potential jar conflicts.
Jongyoul Lee <jo...@gmail.com>于2017年1月12日周四 下午10:18写道：
> Yes, many users suggest that feature to share results between paragraphs
> and different interpreters. I think this would be one of major features in
> a next release.
>
> On Thu, Jan 12, 2017 at 10:30 PM, t p <ta...@gmail.com> wrote:
>
> Is it possible to have similar support to exchange checkbox/dropdown
> variables and can variables be exchanged with other interpreters like PSQL
> (e.g. variable set by spark/pyspark and accessible in another para which is
> running PSQL interpreter).
>
> I’m interested in doing this and I’d like to know if there is a way to
> accomplish this:
>
> https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E
>
>
> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
> There's no way to communicate between spark and sh intepreter. It need to
> implement it but it doesn't yet. But I agree that it would be helpful for
> some cases. Can you create issue?
>
> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <da...@gmail.com>
> wrote:
>
> It's possible to exchange variables between Scala and Spark
> through z.put and z.get.
>
> How to pass a variable to %sh?
>
> In Jupyter it's possible to do for example as
>
> ! hadoop fs -put {localfile} {hdfsfile}
>
>
> where localfile and and hdfsfile are Python variables.
>
> Can't find any references for something similar in Shell Interpreter
> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html
>
> In many notebooks we have to pass small variabels
> from Zeppelin notes to external scripts as parameters.
>
> It would be awesome to have something like
>
> %sh
> /path/to/script --param8={var1} --param9={var2}
>
>
> where var1 and var2 would be implied to be fetched as z.get('var1')
> and z.get('var2') respectively.
>
> Other thoughts?
>
>
> Thank you,
> Ruslan Dautkhanov
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: Passing variables from %pyspark to %sh
Posted by Jongyoul Lee <jo...@gmail.com>.
Yes, many users suggest that feature to share results between paragraphs
and different interpreters. I think this would be one of major features in
a next release.
On Thu, Jan 12, 2017 at 10:30 PM, t p <ta...@gmail.com> wrote:
> Is it possible to have similar support to exchange checkbox/dropdown
> variables and can variables be exchanged with other interpreters like PSQL
> (e.g. variable set by spark/pyspark and accessible in another para which is
> running PSQL interpreter).
>
> I’m interested in doing this and I’d like to know if there is a way to
> accomplish this:
> https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f
> 90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E
>
>
> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
> There's no way to communicate between spark and sh intepreter. It need to
> implement it but it doesn't yet. But I agree that it would be helpful for
> some cases. Can you create issue?
>
> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <da...@gmail.com>
> wrote:
>
>> It's possible to exchange variables between Scala and Spark
>> through z.put and z.get.
>>
>> How to pass a variable to %sh?
>>
>> In Jupyter it's possible to do for example as
>>
>>> ! hadoop fs -put {localfile} {hdfsfile}
>>
>>
>> where localfile and and hdfsfile are Python variables.
>>
>> Can't find any references for something similar in Shell Interpreter
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html
>>
>> In many notebooks we have to pass small variabels
>> from Zeppelin notes to external scripts as parameters.
>>
>> It would be awesome to have something like
>>
>> %sh
>>> /path/to/script --param8={var1} --param9={var2}
>>
>>
>> where var1 and var2 would be implied to be fetched as z.get('var1')
>> and z.get('var2') respectively.
>>
>> Other thoughts?
>>
>>
>> Thank you,
>> Ruslan Dautkhanov
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: Passing variables from %pyspark to %sh
Posted by t p <ta...@gmail.com>.
Is it possible to have similar support to exchange checkbox/dropdown variables and can variables be exchanged with other interpreters like PSQL (e.g. variable set by spark/pyspark and accessible in another para which is running PSQL interpreter).
I’m interested in doing this and I’d like to know if there is a way to accomplish this:
https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E <https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E>
> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
> There's no way to communicate between spark and sh intepreter. It need to implement it but it doesn't yet. But I agree that it would be helpful for some cases. Can you create issue?
>
> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <dautkhanov@gmail.com <ma...@gmail.com>> wrote:
> It's possible to exchange variables between Scala and Spark
> through z.put and z.get.
>
> How to pass a variable to %sh?
>
> In Jupyter it's possible to do for example as
> ! hadoop fs -put {localfile} {hdfsfile}
>
> where localfile and and hdfsfile are Python variables.
>
> Can't find any references for something similar in Shell Interpreter
> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html <https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html>
>
> In many notebooks we have to pass small variabels
> from Zeppelin notes to external scripts as parameters.
>
> It would be awesome to have something like
>
> %sh
> /path/to/script --param8={var1} --param9={var2}
>
> where var1 and var2 would be implied to be fetched as z.get('var1')
> and z.get('var2') respectively.
>
> Other thoughts?
>
>
> Thank you,
> Ruslan Dautkhanov
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>
Re: Passing variables from %pyspark to %sh
Posted by Jongyoul Lee <jo...@gmail.com>.
There's no way to communicate between spark and sh intepreter. It need to
implement it but it doesn't yet. But I agree that it would be helpful for
some cases. Can you create issue?
On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <da...@gmail.com>
wrote:
> It's possible to exchange variables between Scala and Spark
> through z.put and z.get.
>
> How to pass a variable to %sh?
>
> In Jupyter it's possible to do for example as
>
>> ! hadoop fs -put {localfile} {hdfsfile}
>
>
> where localfile and and hdfsfile are Python variables.
>
> Can't find any references for something similar in Shell Interpreter
> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html
>
> In many notebooks we have to pass small variabels
> from Zeppelin notes to external scripts as parameters.
>
> It would be awesome to have something like
>
> %sh
>> /path/to/script --param8={var1} --param9={var2}
>
>
> where var1 and var2 would be implied to be fetched as z.get('var1')
> and z.get('var2') respectively.
>
> Other thoughts?
>
>
> Thank you,
> Ruslan Dautkhanov
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net