[jira] [Created] (TOREE-460) Difference in output between Spark 2.1.0 and Spark 2.1.2+

["Vinayak Joshi (JIRA)" <ji...@apache.org>]

Vinayak Joshi created TOREE-460:
-----------------------------------

             Summary: Difference in output between Spark 2.1.0 and Spark 2.1.2+
                 Key: TOREE-460
                 URL: https://issues.apache.org/jira/browse/TOREE-460
             Project: TOREE
          Issue Type: Bug
          Components: Kernel
    Affects Versions: 0.2.0
            Reporter: Vinayak Joshi


Using the current version of Toree (0.2.0) & Scala 2.11 when Scala code such as this is executed:

```
var v = "abc"
```

against Spark 2.1.0, there is no output generated on the notebook. However, against Spark 2.1.2 and above, the notebook displays the following lines as output for the same code:

```
v = abc
Out[2]: abc
```

When used via a kernel-gateway, I could trace the websocket messages and observed that a couple of extra messages are received by the notebook when the code is executed against Spark 2.1.2+ :

```
  {
  	"parent_header": {
           ...
  	},
  	"msg_type": "display_data",
  	"msg_id": "7ee0fe40-3283-45dc-8632-a9f013ae7052",
  	"content": {
  		"source": "user",
  		"data": {
  			"text/plain": "v = abc"
  		},
  		"metadata": {}
  	},
  	"header": {
           ...
  	},
  	"channel": "iopub",
  	"buffers": [],
  	"metadata": {
  		"timestamp": "1513154854626"
  	}
  }
```

and

```
  {
  	"parent_header": {
           ...
  	},
  	"msg_type": "execute_result",
  	"msg_id": "98dae777-c5cf-4ae7-9903-306e1fdf7bc1",
  	"content": {
  		"execution_count": 2,
  		"data": {
  			"text/plain": "abc"
  		},
  		"metadata": {}
  	},
  	"header": {
           ...
  	},
  	"channel": "iopub",
  	"buffers": [],
  	"metadata": {
  		"timestamp": "1513154854754"
  	}
  }
```
These messages appear to be getting rendered on the notebook ui. 

This is breaking consistency/compatibility for expected outputs when Spark version changes for notebooks. Also there are users that use some kind of code cell hiding to prevent credentials etc being displayed in a notebook - this use case now breaks since the cell output now displays any credential values that are assigned to variables. 

So looks like something's changed between Spark 2.1.0 and later versions that causes Toree to produce different replies to the same code being executed? Is it something Toree needs to adapt to?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)