Ipython notebook, ec2 spark cluster and matplotlib

[Marco Didonna <m....@gmail.com>]

Hello everybody,
I'm running a two node spark cluster on ec2, created using the provided
scripts. I then ssh into the master and invoke
"PYSPARK_DRIVER_PYTHON=ipython  PYSPARK_DRIVER_PYTHON_OPTS='notebook
--profile=pyspark' spark/bin/pyspark". This launches a spark notebook which
has been instructed to listen to all interfaces, not only localhost. I then
open my browser and start playing around.

All commands run fine as far as I've seen but there's an annoying problem:
I cannot display matplotlib graphs in a cell, I get the following error
"TclError: no display name and no $DISPLAY environment variable".

I've searched the web and I've tried the following two approaches:

1. use -X to enable X11 forwarding: when I use this option I get no error,
a slow execution time and no image at all

2. use matplotlib.use('agg'), no image but if I execute fig.savefig I can
totally see the image being created.

Has anybody have a similar problem? If so can you help me troubleshoot?

Thanks,
MD