[jira] [Reopened] (TOREE-460) Difference in output with Toree builds containing fix for TOREE-380

["Vinayak Joshi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TOREE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinayak Joshi reopened TOREE-460:
---------------------------------

I reported the issue initially as caused by difference in Spark versions, but it turns out the root cause was a difference in the Toree build the env used. The observations noted already in terms of difference in output and the extra WebSocket messages remains valid. Only that this change occurs beginning with this commit - https://github.com/apache/incubator-toree/commit/5cd1727089eff26f553f990ab9dfe49d2be14fe2 (TOREE-380: Allow interpreters to produce output by MIME type.)

> Difference in output with Toree builds containing fix for TOREE-380
> -------------------------------------------------------------------
>
>                 Key: TOREE-460
>                 URL: https://issues.apache.org/jira/browse/TOREE-460
>             Project: TOREE
>          Issue Type: Bug
>          Components: Kernel
>    Affects Versions: 0.2.0
>            Reporter: Vinayak Joshi
>            Assignee: Luciano Resende
>             Fix For: 0.2.0
>
>
> Using the current version of Toree (0.2.0) & Scala 2.11 when Scala code such as this is executed:
> ```
> var v = "abc"
> ```
> against Spark 2.1.0, there is no output generated on the notebook. However, against Spark 2.1.2 and above, the notebook displays the following lines as output for the same code:
> ```
> v = abc
> Out[2]: abc
> ```
> When used via a kernel-gateway, I could trace the websocket messages and observed that a couple of extra messages are received by the notebook when the code is executed against Spark 2.1.2+ :
> ```
>   {
>   	"parent_header": {
>            ...
>   	},
>   	"msg_type": "display_data",
>   	"msg_id": "7ee0fe40-3283-45dc-8632-a9f013ae7052",
>   	"content": {
>   		"source": "user",
>   		"data": {
>   			"text/plain": "v = abc"
>   		},
>   		"metadata": {}
>   	},
>   	"header": {
>            ...
>   	},
>   	"channel": "iopub",
>   	"buffers": [],
>   	"metadata": {
>   		"timestamp": "1513154854626"
>   	}
>   }
> ```
> and
> ```
>   {
>   	"parent_header": {
>            ...
>   	},
>   	"msg_type": "execute_result",
>   	"msg_id": "98dae777-c5cf-4ae7-9903-306e1fdf7bc1",
>   	"content": {
>   		"execution_count": 2,
>   		"data": {
>   			"text/plain": "abc"
>   		},
>   		"metadata": {}
>   	},
>   	"header": {
>            ...
>   	},
>   	"channel": "iopub",
>   	"buffers": [],
>   	"metadata": {
>   		"timestamp": "1513154854754"
>   	}
>   }
> ```
> These messages appear to be getting rendered on the notebook ui. 
> This is breaking consistency/compatibility for expected outputs when Spark version changes for notebooks. Also there are users that use some kind of code cell hiding to prevent credentials etc being displayed in a notebook - this use case now breaks since the cell output now displays any credential values that are assigned to variables. 
> So looks like something's changed between Spark 2.1.0 and later versions that causes Toree to produce different replies to the same code being executed? Is it something Toree needs to adapt to?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)