[jira] [Comment Edited] (LIVY-507) Livy 0.5 incubating. Dumps output of command run in the browser.

["Glenn McCall (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611362#comment-16611362 ] 

Glenn McCall edited comment on LIVY-507 at 9/11/18 11:28 PM:
-------------------------------------------------------------

Hi Saiso,

I understand what the change is and how it faithfully reflects the output given in Scala REPL.

However, IMHO, I do not see Zeppelin as a Scala REPL. Zeppelin is more of a notebook that has a similar look and feel to a WIki - the big feature is the ability to have live code that produces nice output for perusal by end users and in our case, and I'm sure many others presentation to non-technical users such as management. When looking at a Wiki page you are not exposed to the Gory details of HTML, CSS and so on - you see the final formatted output. I feel that Zeppelin should be the same - you should be able to see the final formatted output without explicitly having to hide the gory intermediate REPL outputs via a series of manual efforts that must be performed for each and every paragraph in each and every notebook. 

Faithfully capturing the entire REPL output is clever, but it messes up the notebook.
 IMHO, if you want to see REPL output, then use a REPL such as SBT, Spark-Shell, IntelliJ etc. There are other advantages to using a real REPL as you get context sensitive "code completion" in the REPL.

While I can understand the need, maybe, for having debugging output which shows the entire REPL output so you can see what is going on, there should at least be a method (under control of the notebook code) that *manually turns this on*. There should, IMHO, be backward comparability, so, show the output of a paragraph as it was in 0.4 (i.e. only the last statement output is shown) and an *option*, which can manually be turned on to *dump* the entire REPL output if required. Just as with a Wiki page, if you really need to see the HTML, CSS, Javascript and so on, you can manually show the page source within the browser.

The proposed workarounds to hide the REPL output such as:
 * splitting a paragraph into two being a) preparatory piece of code - in which code and output are hidden and b) a single println statement (or as per the justification in the original Jira) two print statements in a single paragraph
 * Enclosing the entire code segment in to a block (i.e. putting braces around it)
 * Others
 Create their own problems IMHO

For example:

Splitting the paragraph:
 * If you wish to refresh your output, you have to remember to run two paragraphs - the preparatory code and the display code (and you have to train your users to do that as well - previously you could say if you want to refresh this chart, just click it's run button - now you have to tell them to click it's run button, but first click this seemingly unrelated other run button(s)).
 * The extra paragraphs containing preparatory code consume some screen real estate. If, like me, you want to modularise and have paragraphs for specific purposes, you will end up with several paragraphs consuming screen real estate just for the setup code.
 * Whether you modularise or not, it is possible that you will end up with large paragraphs (e.g. 40-50 lines of scala code). If you have a runtime error it can often be difficult to see because:
 ** Previously, when only the last output was displayed, the exception will appear *at the top* of the output area of the paragraph. 
 ** Now, when all REPL output is displayed it can be difficult to tell if your code has a runtime error as it will often be off the bottom of the output area (or off the bottom of the browser). And if the little "Status indicator" next to the run paragraph button is not visible (i.e. you run the paragraph via the keyboard shortcuts) it can be difficult to just see whether there is a problem or not.
 * Password management - we need to secure passwords from casual observation. Part of our process involves decryption password for use with external systems. When all of the REPL output is captured and displayed an embedded line of code faithfully shows our passwords in cleartext - this is highly undesirable. Sure, we can hide the output, but if for some reason we need to expose it (e.g. to view the runtime error) then the password is their for everyone to see - or we need more paragraphs (e.g. modularise the password out into its own paragraph) which occupy more screen real estates (there are other reasons why this is not a good idea for the password management use case)

Enclosing the code in a block:
 * This is a nice workaround, but it requires refactoring all of our code
 * Variables defined in the code block are not visible to subsequent paragraphs as they become out of scope at the end of the code block/paragraph defining them.
 Thus this has some usage, but has limited use cases (I have several notebooks that already work this way)

 

Concluding:
 * The supporting use case is, IMHO, an extreme edge case. The supporting use case was something like "Print(1)\n println(2)" results in only 2 being displayed when the desired output is 12.
 However, consider the following more realistic use case:

{code:java}
println("result:")
val x = 1
val y = 2
println(s"x + y = ${x + y}")

{code}
Which will result in:
{code:java}
result:
x: Int = 1
y: Int = 2
x + y = 3

{code}
So you still do not get the "nice output" proposed in the documentation supporting the use case.

So, you might argue split the paragraphs and just have:
{code:java}
val x = 1
val y = 2
{code}
followed by:
{code:java}
println("result:")
println(s"x + y = ${x + y}")

{code}
To which I would reply just merge the two print statements into one using the plus operator e.g. println("result:\n" + s"x + y = ${x + y}") or more simply println(s"result\nx + y = ${x + y}")

No matter how you re-arrange the initial example into a single paragraph, you will still get messy output:

e.g.
{code:java}
val x = 1
val y = 2
print("result")
println(s"x + y = ${x + y}")
{code}
Produces:
{code:java}
x: Int = 1
y: Int = 2
result:
x + y = 3

{code}
which IMHO, is not much better


was (Author: gm310509):
Hi Saiso,

I understand what the change is and how it faithfully reflects the output given in Scala REPL.

However, IMHO, I do not see Zeppelin as a Scala REPL. Zeppelin is more of a notebook that has a similar look and feel to a WIki - the big feature is the ability to have live code that produces nice output for perusal by end users and in our case, and I'm sure many others presentation to non-technical users such as management.

Faithfully capturing the entire REPL output is clever, but it messes up the notebook.
IMHO, if you want to see REPL output, use a REPL such as SBT, Spark-Shell, IntelliJ etc. There are advantages to using a real REPL as you get context sensitive "code completion" in the REPL.

The proposed workarounds such as:
 * splitting a paragraph into two being a) preparatory piece of code - in which code and output are hidden and b) a single println statement (or as per the justification in the original Jira) two print statements in a single paragraph
 * Enclosing the entire code segment in to a block (i.e. putting braces around it)
 * Others
Create their own problems IMHO

For example:

Splitting the paragraph:
 * If you wish to refresh your output, you have to remember to run two paragraphs - the preparatory code and the display code.
 * The extra paragraphs containing preparatory code consume some screen real estate. If, like me, you want to modularise and have paragraphs for specific purposes, you will end up with several paragraphs consuming screen real estate just for the setup code.
 * Whether you modularise or not, it is possible that you will end up with large paragraphs (e.g. 40-50 lines of scala code). If you have a runtime error it can often be difficult to see because:
 ** Previously, when only the last output was displayed, the exception will appear *at the top* of the output area of the paragraph. 
 ** Now, when all REPL output is displayed it can be difficult to tell if your code has a runtime error as it will often be off the bottom of the output area (or off the bottom of the browser). And if the little "Status indicator" next to the run paragraph button is not visible (i.e. you run the paragraph via the keyboard shortcuts) it can be difficult to just see whether there is a problem or not.
 * Password management - we need to secure passwords from casual observation. Part of our process involves decryption password for use with external systems. When all of the REPL output is captured and displayed an embedded line of code faithfully shows our passwords in cleartext - this is highly undesirable. Sure, we can hide the output, but if for some reason we need to expose it (e.g. to view the runtime error) then the password is their for everyone to see - or we need more paragraphs (e.g. modularise the password out into its own paragraph) which occupy more screen real estates (there are other reasons why this is not a good idea for the password management use case)

Enclosing the code in a block:
 * This is a nice workaround, but it requires refactoring all of our code
 * Variables defined in the code block are not visible to subsequent paragraphs as they become out of scope at the end of the code block/paragraph defining them.
Thus this has some usage, but has limited use cases (I have several notebooks that already work this way)

 

Concluding:
 * The supporting use case is, IMHO, an extreme edge case. The supporting use case was something like "Print(1)\n println(2)" results in only 2 being displayed when the desired output is 12.
However, consider the following more realistic use case:
 * 
{code:java}
println("1")
{code}

 * 
 * 

 

> Livy 0.5 incubating. Dumps output of command run in the browser.
> ----------------------------------------------------------------
>
>                 Key: LIVY-507
>                 URL: https://issues.apache.org/jira/browse/LIVY-507
>             Project: Livy
>          Issue Type: Bug
>          Components: Interpreter
>    Affects Versions: 0.5.0
>            Reporter: Bicky
>            Priority: Blocker
>             Fix For: 0.5.0
>
>         Attachments: Screen Shot 2018-09-10 at 4.57.18 pm.png
>
>
> Team,
> We upgraded to livy 0.5 incubating post which we are seeing that livy interpreter on zeppelin is displaying the code of the paragraph  in the browser itself .
>  
> !Screen Shot 2018-09-10 at 4.57.18 pm.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)