You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Patel <ar...@gmail.com> on 2016/07/03 22:48:10 UTC
Graphframe Error
I started my pyspark shell with command (I am using spark 1.6).
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
I have copied
http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar
to the lib directory of Spark as well.
I was getting below error
>>> from graphframes import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zipimport.ZipImportError: can't find module 'graphframes'
>>>
So, as per suggestions from similar questions, I have extracted the
graphframes python directory and copied to the local directory where I am
running pyspark.
>>> from graphframes import *
But, not able to create the GraphFrame
>>> g = GraphFrame(v, e)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'GraphFrame' is not defined
Also, I am getting below error.
>>> from graphframes.examples import Graphs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Bad magic number in graphframes/examples.pyc
Any help will be highly appreciated.
- Arun
Re: Graphframe Error
Posted by Felix Cheung <fe...@hotmail.com>.
I ran it with Python 2.
On Thu, Jul 7, 2016 at 4:13 AM -0700, "Arun Patel" <ar...@gmail.com>> wrote:
I have tied this already. It does not work.
What version of Python is needed for this package?
On Wed, Jul 6, 2016 at 12:45 AM, Felix Cheung <fe...@hotmail.com>> wrote:
This could be the workaround:
http://stackoverflow.com/a/36419857
On Tue, Jul 5, 2016 at 5:37 AM -0700, "Arun Patel" <ar...@gmail.com>> wrote:
Thanks Yanbo and Felix.
I tried these commands on CDH Quickstart VM and also on "Spark 1.6 pre-built for Hadoop" version. I am still not able to get it working. Not sure what I am missing. Attaching the logs.
On Mon, Jul 4, 2016 at 5:33 AM, Felix Cheung <fe...@hotmail.com>> wrote:
It looks like either the extracted Python code is corrupted or there is a mismatch Python version. Are you using Python 3?
stackoverflow.com/questions/514371/whats-the-bad-magic-number-error<http://stackoverflow.com/questions/514371/whats-the-bad-magic-number-error>
On Mon, Jul 4, 2016 at 1:37 AM -0700, "Yanbo Liang" <yb...@gmail.com>> wrote:
Hi Arun,
The command
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
will automatically load the required graphframes jar file from maven repository, it was not affected by the location where the jar file was placed. Your examples works well in my laptop.
Or you can use try with
bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
to launch PySpark with graphframes enabled. You should set "--py-files" and "--jars" options with the directory where you saved graphframes.jar.
Thanks
Yanbo
2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>>:
I started my pyspark shell with command (I am using spark 1.6).
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
I have copied http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar to the lib directory of Spark as well.
I was getting below error
>>> from graphframes import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zipimport.ZipImportError: can't find module 'graphframes'
>>>
So, as per suggestions from similar questions, I have extracted the graphframes python directory and copied to the local directory where I am running pyspark.
>>> from graphframes import *
But, not able to create the GraphFrame
>>> g = GraphFrame(v, e)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'GraphFrame' is not defined
Also, I am getting below error.
>>> from graphframes.examples import Graphs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Bad magic number in graphframes/examples.pyc
Any help will be highly appreciated.
- Arun
Re: Graphframe Error
Posted by Arun Patel <ar...@gmail.com>.
I have tied this already. It does not work.
What version of Python is needed for this package?
On Wed, Jul 6, 2016 at 12:45 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> This could be the workaround:
>
> http://stackoverflow.com/a/36419857
>
>
>
>
> On Tue, Jul 5, 2016 at 5:37 AM -0700, "Arun Patel" <
> arunp.bigdata@gmail.com> wrote:
>
> Thanks Yanbo and Felix.
>
> I tried these commands on CDH Quickstart VM and also on "Spark 1.6
> pre-built for Hadoop" version. I am still not able to get it working. Not
> sure what I am missing. Attaching the logs.
>
>
>
>
> On Mon, Jul 4, 2016 at 5:33 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> It looks like either the extracted Python code is corrupted or there is a
>> mismatch Python version. Are you using Python 3?
>>
>>
>> stackoverflow.com/questions/514371/whats-the-bad-magic-number-error
>>
>>
>>
>>
>>
>> On Mon, Jul 4, 2016 at 1:37 AM -0700, "Yanbo Liang" <yb...@gmail.com>
>> wrote:
>>
>> Hi Arun,
>>
>> The command
>>
>> bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
>>
>> will automatically load the required graphframes jar file from maven
>> repository, it was not affected by the location where the jar file was
>> placed. Your examples works well in my laptop.
>>
>> Or you can use try with
>>
>> bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
>>
>> to launch PySpark with graphframes enabled. You should set "--py-files"
>> and "--jars" options with the directory where you saved graphframes.jar.
>>
>> Thanks
>> Yanbo
>>
>>
>> 2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>:
>>
>>> I started my pyspark shell with command (I am using spark 1.6).
>>>
>>> bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
>>>
>>> I have copied
>>> http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar
>>> to the lib directory of Spark as well.
>>>
>>> I was getting below error
>>>
>>> >>> from graphframes import *
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> zipimport.ZipImportError: can't find module 'graphframes'
>>> >>>
>>>
>>> So, as per suggestions from similar questions, I have extracted the
>>> graphframes python directory and copied to the local directory where I am
>>> running pyspark.
>>>
>>> >>> from graphframes import *
>>>
>>> But, not able to create the GraphFrame
>>>
>>> >>> g = GraphFrame(v, e)
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> NameError: name 'GraphFrame' is not defined
>>>
>>> Also, I am getting below error.
>>> >>> from graphframes.examples import Graphs
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> ImportError: Bad magic number in graphframes/examples.pyc
>>>
>>> Any help will be highly appreciated.
>>>
>>> - Arun
>>>
>>
>>
>
Re: Graphframe Error
Posted by Felix Cheung <fe...@hotmail.com>.
This could be the workaround:
http://stackoverflow.com/a/36419857
On Tue, Jul 5, 2016 at 5:37 AM -0700, "Arun Patel" <ar...@gmail.com>> wrote:
Thanks Yanbo and Felix.
I tried these commands on CDH Quickstart VM and also on "Spark 1.6 pre-built for Hadoop" version. I am still not able to get it working. Not sure what I am missing. Attaching the logs.
On Mon, Jul 4, 2016 at 5:33 AM, Felix Cheung <fe...@hotmail.com>> wrote:
It looks like either the extracted Python code is corrupted or there is a mismatch Python version. Are you using Python 3?
stackoverflow.com/questions/514371/whats-the-bad-magic-number-error<http://stackoverflow.com/questions/514371/whats-the-bad-magic-number-error>
On Mon, Jul 4, 2016 at 1:37 AM -0700, "Yanbo Liang" <yb...@gmail.com>> wrote:
Hi Arun,
The command
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
will automatically load the required graphframes jar file from maven repository, it was not affected by the location where the jar file was placed. Your examples works well in my laptop.
Or you can use try with
bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
to launch PySpark with graphframes enabled. You should set "--py-files" and "--jars" options with the directory where you saved graphframes.jar.
Thanks
Yanbo
2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>>:
I started my pyspark shell with command (I am using spark 1.6).
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
I have copied http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar to the lib directory of Spark as well.
I was getting below error
>>> from graphframes import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zipimport.ZipImportError: can't find module 'graphframes'
>>>
So, as per suggestions from similar questions, I have extracted the graphframes python directory and copied to the local directory where I am running pyspark.
>>> from graphframes import *
But, not able to create the GraphFrame
>>> g = GraphFrame(v, e)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'GraphFrame' is not defined
Also, I am getting below error.
>>> from graphframes.examples import Graphs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Bad magic number in graphframes/examples.pyc
Any help will be highly appreciated.
- Arun
Re: Graphframe Error
Posted by Arun Patel <ar...@gmail.com>.
Thanks Yanbo and Felix.
I tried these commands on CDH Quickstart VM and also on "Spark 1.6
pre-built for Hadoop" version. I am still not able to get it working. Not
sure what I am missing. Attaching the logs.
On Mon, Jul 4, 2016 at 5:33 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> It looks like either the extracted Python code is corrupted or there is a
> mismatch Python version. Are you using Python 3?
>
>
> stackoverflow.com/questions/514371/whats-the-bad-magic-number-error
>
>
>
>
>
> On Mon, Jul 4, 2016 at 1:37 AM -0700, "Yanbo Liang" <yb...@gmail.com>
> wrote:
>
> Hi Arun,
>
> The command
>
> bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
>
> will automatically load the required graphframes jar file from maven
> repository, it was not affected by the location where the jar file was
> placed. Your examples works well in my laptop.
>
> Or you can use try with
>
> bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
>
> to launch PySpark with graphframes enabled. You should set "--py-files"
> and "--jars" options with the directory where you saved graphframes.jar.
>
> Thanks
> Yanbo
>
>
> 2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>:
>
>> I started my pyspark shell with command (I am using spark 1.6).
>>
>> bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
>>
>> I have copied
>> http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar
>> to the lib directory of Spark as well.
>>
>> I was getting below error
>>
>> >>> from graphframes import *
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> zipimport.ZipImportError: can't find module 'graphframes'
>> >>>
>>
>> So, as per suggestions from similar questions, I have extracted the
>> graphframes python directory and copied to the local directory where I am
>> running pyspark.
>>
>> >>> from graphframes import *
>>
>> But, not able to create the GraphFrame
>>
>> >>> g = GraphFrame(v, e)
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> NameError: name 'GraphFrame' is not defined
>>
>> Also, I am getting below error.
>> >>> from graphframes.examples import Graphs
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> ImportError: Bad magic number in graphframes/examples.pyc
>>
>> Any help will be highly appreciated.
>>
>> - Arun
>>
>
>
Re: Graphframe Error
Posted by Felix Cheung <fe...@hotmail.com>.
It looks like either the extracted Python code is corrupted or there is a mismatch Python version. Are you using Python 3?
stackoverflow.com/questions/514371/whats-the-bad-magic-number-error<http://stackoverflow.com/questions/514371/whats-the-bad-magic-number-error>
On Mon, Jul 4, 2016 at 1:37 AM -0700, "Yanbo Liang" <yb...@gmail.com>> wrote:
Hi Arun,
The command
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
will automatically load the required graphframes jar file from maven repository, it was not affected by the location where the jar file was placed. Your examples works well in my laptop.
Or you can use try with
bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
to launch PySpark with graphframes enabled. You should set "--py-files" and "--jars" options with the directory where you saved graphframes.jar.
Thanks
Yanbo
2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>>:
I started my pyspark shell with command (I am using spark 1.6).
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
I have copied http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar to the lib directory of Spark as well.
I was getting below error
>>> from graphframes import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zipimport.ZipImportError: can't find module 'graphframes'
>>>
So, as per suggestions from similar questions, I have extracted the graphframes python directory and copied to the local directory where I am running pyspark.
>>> from graphframes import *
But, not able to create the GraphFrame
>>> g = GraphFrame(v, e)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'GraphFrame' is not defined
Also, I am getting below error.
>>> from graphframes.examples import Graphs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Bad magic number in graphframes/examples.pyc
Any help will be highly appreciated.
- Arun
Re: Graphframe Error
Posted by Yanbo Liang <yb...@gmail.com>.
Hi Arun,
The command
bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
will automatically load the required graphframes jar file from maven
repository, it was not affected by the location where the jar file was
placed. Your examples works well in my laptop.
Or you can use try with
bin/pyspark --py-files ***/graphframes.jar --jars ***/graphframes.jar
to launch PySpark with graphframes enabled. You should set "--py-files" and
"--jars" options with the directory where you saved graphframes.jar.
Thanks
Yanbo
2016-07-03 15:48 GMT-07:00 Arun Patel <ar...@gmail.com>:
> I started my pyspark shell with command (I am using spark 1.6).
>
> bin/pyspark --packages graphframes:graphframes:0.1.0-spark1.6
>
> I have copied
> http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.1.0-spark1.6/graphframes-0.1.0-spark1.6.jar
> to the lib directory of Spark as well.
>
> I was getting below error
>
> >>> from graphframes import *
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> zipimport.ZipImportError: can't find module 'graphframes'
> >>>
>
> So, as per suggestions from similar questions, I have extracted the
> graphframes python directory and copied to the local directory where I am
> running pyspark.
>
> >>> from graphframes import *
>
> But, not able to create the GraphFrame
>
> >>> g = GraphFrame(v, e)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> NameError: name 'GraphFrame' is not defined
>
> Also, I am getting below error.
> >>> from graphframes.examples import Graphs
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: Bad magic number in graphframes/examples.pyc
>
> Any help will be highly appreciated.
>
> - Arun
>