You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by Ia...@tdameritrade.com on 2016/11/02 19:22:08 UTC
matplotlib in Toree
Hi,
I just noticed that simple plots with matplotlib do not work in Toree. I get this error in the UI:
Magic pyspark failed to execute with error:
null was reset!
In the logs I see:
16/11/02 14:12:15 ERROR PySparkProcessHandler: null process failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 134 (Exit value: 134)
16/11/02 14:12:15 WARN MagicManager: Magic pyspark failed to execute with error:
null was reset!
res12: Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput] =
Left(Map(text/plain -> Magic pyspark failed to execute with error:
null was reset!))
Is there a fix for this or some kind of work around? Otherwise, if you let me know what issue type, I can create an issue in Jira.
Thanks,
Ian
Re: matplotlib in Toree
Posted by Ia...@tdameritrade.com.
Thanks Marius!
This ended up working in python:
kernel.display().html(³some html²)
On 11/2/16, 7:46 PM, "Marius van Niekerk" <ma...@gmail.com>
wrote:
>So matplotlib isn't very well supported at the moment. Toree's pyspark
>kernel is not ipykernel -- so the magics that work with that (and
>matplotlib integration) sadly won't work.
>
>You can do it by calling the kernel.display methods afaik.
>On Wed, Nov 2, 2016 at 15:22 <Ia...@tdameritrade.com> wrote:
>
>> Hi,
>>
>> I just noticed that simple plots with matplotlib do not work in Toree. I
>> get this error in the UI:
>>
>>
>> Magic pyspark failed to execute with error:
>> null was reset!
>>
>>
>> In the logs I see:
>>
>> 16/11/02 14:12:15 ERROR PySparkProcessHandler: null process failed:
>> org.apache.commons.exec.ExecuteException: Process exited with an error:
>>134
>> (Exit value: 134)
>> 16/11/02 14:12:15 WARN MagicManager: Magic pyspark failed to execute
>>with
>> error:
>> null was reset!
>> res12:
>>
>>Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.Line
>>MagicOutput]
>> =
>> Left(Map(text/plain -> Magic pyspark failed to execute with error:
>> null was reset!))
>>
>> Is there a fix for this or some kind of work around? Otherwise, if you
>>let
>> me know what issue type, I can create an issue in Jira.
>>
>> Thanks,
>>
>> Ian
>>
>--
>regards
>Marius van Niekerk
Re: matplotlib in Toree
Posted by Ia...@tdameritrade.com.
I¹ve been having some issues with this logic. Once I get to sizable
datasets (~ 40,000), it no longer works, while with smaller ones ~1000 it
works. I¹m thinking the HTML becomes too much, since this works fine using
seaborn/matplotlib alone:
%%pyspark
# seaborn stuff
sns.set(style='darkgrid')
sm = sns.PairGrid(df_sub.tail(150),
hue='label', palette={0: '#931D3D', 1: '#54B847'},
size=4, hue_kws={'marker': ['o', 'D']})
sm = sm.map_offdiag(plt.scatter, alpha=0.25, s=25)
sm = sm.map_diag(plt.hist, bins=10)
sm = sm.add_legend()
# trying to get it to show up
img = StringIO.StringIO()
sm.savefig(img, format='svg')
img.seek(0)
html_txt = "<div style='width:600px'>" + img.buf + "</div>"
kernel.display().html(html_txt)
Any ideas how to get that second block leveraging Toree in a better way?
Thanks,
Ian
On 11/2/16, 7:46 PM, "Marius van Niekerk" <ma...@gmail.com>
wrote:
>So matplotlib isn't very well supported at the moment. Toree's pyspark
>kernel is not ipykernel -- so the magics that work with that (and
>matplotlib integration) sadly won't work.
>
>You can do it by calling the kernel.display methods afaik.
>On Wed, Nov 2, 2016 at 15:22 <Ia...@tdameritrade.com> wrote:
>
>> Hi,
>>
>> I just noticed that simple plots with matplotlib do not work in Toree. I
>> get this error in the UI:
>>
>>
>> Magic pyspark failed to execute with error:
>> null was reset!
>>
>>
>> In the logs I see:
>>
>> 16/11/02 14:12:15 ERROR PySparkProcessHandler: null process failed:
>> org.apache.commons.exec.ExecuteException: Process exited with an error:
>>134
>> (Exit value: 134)
>> 16/11/02 14:12:15 WARN MagicManager: Magic pyspark failed to execute
>>with
>> error:
>> null was reset!
>> res12:
>>
>>Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.Line
>>MagicOutput]
>> =
>> Left(Map(text/plain -> Magic pyspark failed to execute with error:
>> null was reset!))
>>
>> Is there a fix for this or some kind of work around? Otherwise, if you
>>let
>> me know what issue type, I can create an issue in Jira.
>>
>> Thanks,
>>
>> Ian
>>
>--
>regards
>Marius van Niekerk
Re: matplotlib in Toree
Posted by Marius van Niekerk <ma...@gmail.com>.
So matplotlib isn't very well supported at the moment. Toree's pyspark
kernel is not ipykernel -- so the magics that work with that (and
matplotlib integration) sadly won't work.
You can do it by calling the kernel.display methods afaik.
On Wed, Nov 2, 2016 at 15:22 <Ia...@tdameritrade.com> wrote:
> Hi,
>
> I just noticed that simple plots with matplotlib do not work in Toree. I
> get this error in the UI:
>
>
> Magic pyspark failed to execute with error:
> null was reset!
>
>
> In the logs I see:
>
> 16/11/02 14:12:15 ERROR PySparkProcessHandler: null process failed:
> org.apache.commons.exec.ExecuteException: Process exited with an error: 134
> (Exit value: 134)
> 16/11/02 14:12:15 WARN MagicManager: Magic pyspark failed to execute with
> error:
> null was reset!
> res12:
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
> =
> Left(Map(text/plain -> Magic pyspark failed to execute with error:
> null was reset!))
>
> Is there a fix for this or some kind of work around? Otherwise, if you let
> me know what issue type, I can create an issue in Jira.
>
> Thanks,
>
> Ian
>
--
regards
Marius van Niekerk