[jira] [Comment Edited] (TOREE-460) Difference in output with Toree builds containing fix for TOREE-380

["Vinayak Joshi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16296421#comment-16296421 ] 

Vinayak Joshi edited comment on TOREE-460 at 12/19/17 8:06 AM:
---------------------------------------------------------------

I reported the issue initially as caused by difference in Spark versions, but it turns out the root cause was a difference in the Toree build the env used. The observations noted already in terms of difference in output and the extra WebSocket messages remains valid. Only that this change occurs beginning with this commit - https://github.com/apache/incubator-toree/commit/5cd1727089eff26f553f990ab9dfe49d2be14fe2 (TOREE-380: Allow interpreters to produce output by MIME type.)

Builds prior to this commit seem to suppress the output, but with later builds, the notebook ends up displaying the output given above.


was (Author: vijoshi):
I reported the issue initially as caused by difference in Spark versions, but it turns out the root cause was a difference in the Toree build the env used. The observations noted already in terms of difference in output and the extra WebSocket messages remains valid. Only that this change occurs beginning with this commit - https://github.com/apache/incubator-toree/commit/5cd1727089eff26f553f990ab9dfe49d2be14fe2 (TOREE-380: Allow interpreters to produce output by MIME type.)

> Difference in output with Toree builds containing fix for TOREE-380
> -------------------------------------------------------------------
>
>                 Key: TOREE-460
>                 URL: https://issues.apache.org/jira/browse/TOREE-460
>             Project: TOREE
>          Issue Type: Bug
>          Components: Kernel
>    Affects Versions: 0.2.0
>            Reporter: Vinayak Joshi
>            Assignee: Luciano Resende
>             Fix For: 0.2.0
>
>
> Using the current version of Toree (0.2.0) & Scala 2.11 when Scala code such as this is executed:
> ```
> var v = "abc"
> ```
> against Spark 2.1.0, there is no output generated on the notebook. However, against Spark 2.1.2 and above, the notebook displays the following lines as output for the same code:
> ```
> v = abc
> Out[2]: abc
> ```
> When used via a kernel-gateway, I could trace the websocket messages and observed that a couple of extra messages are received by the notebook when the code is executed against Spark 2.1.2+ :
> ```
>   {
>   	"parent_header": {
>            ...
>   	},
>   	"msg_type": "display_data",
>   	"msg_id": "7ee0fe40-3283-45dc-8632-a9f013ae7052",
>   	"content": {
>   		"source": "user",
>   		"data": {
>   			"text/plain": "v = abc"
>   		},
>   		"metadata": {}
>   	},
>   	"header": {
>            ...
>   	},
>   	"channel": "iopub",
>   	"buffers": [],
>   	"metadata": {
>   		"timestamp": "1513154854626"
>   	}
>   }
> ```
> and
> ```
>   {
>   	"parent_header": {
>            ...
>   	},
>   	"msg_type": "execute_result",
>   	"msg_id": "98dae777-c5cf-4ae7-9903-306e1fdf7bc1",
>   	"content": {
>   		"execution_count": 2,
>   		"data": {
>   			"text/plain": "abc"
>   		},
>   		"metadata": {}
>   	},
>   	"header": {
>            ...
>   	},
>   	"channel": "iopub",
>   	"buffers": [],
>   	"metadata": {
>   		"timestamp": "1513154854754"
>   	}
>   }
> ```
> These messages appear to be getting rendered on the notebook ui. 
> This is breaking consistency/compatibility for expected outputs when Spark version changes for notebooks. Also there are users that use some kind of code cell hiding to prevent credentials etc being displayed in a notebook - this use case now breaks since the cell output now displays any credential values that are assigned to variables. 
> So looks like something's changed between Spark 2.1.0 and later versions that causes Toree to produce different replies to the same code being executed? Is it something Toree needs to adapt to?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)