You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by ping yan <sh...@gmail.com> on 2015/10/09 07:28:17 UTC

weird issue with sqlContext.createDataFrame - pyspark 1.3.1

I really cannot figure out what this is about..
(tried to import pandas, in case that is a dependency, but it didn't help.)

>>> from pyspark.sql import SQLContext
>>> sqlContext=SQLContext(sc)
>>> sqlContext.createDataFrame(l).collect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/python/pyspark/sql/context.py",
line 318, in createDataFrame
    if has_pandas and isinstance(data, pandas.DataFrame):
AttributeError: 'module' object has no attribute 'DataFrame'

Would appreciate any pointers.

Thanks!
Ping

Re: weird issue with sqlContext.createDataFrame - pyspark 1.3.1

Posted by ping yan <sh...@gmail.com>.

Thanks. It does seem like that my pandas installation is corrupted.

Thanks!


On Fri, Oct 9, 2015 at 11:04 AM, Davies Liu <da...@databricks.com> wrote:

> Is it possible that you have an very old version of pandas, that does
> not have DataFrame (or in different submodule).
>
> Could you try this:
> ```
> >>> import pandas
> >>> pandas.__version__
> '0.14.0'
> ```
>
> On Thu, Oct 8, 2015 at 10:28 PM, ping yan <sh...@gmail.com> wrote:
> > I really cannot figure out what this is about..
> > (tried to import pandas, in case that is a dependency, but it didn't
> help.)
> >
> >>>> from pyspark.sql import SQLContext
> >>>> sqlContext=SQLContext(sc)
> >>>> sqlContext.createDataFrame(l).collect()
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File
> >
> "/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/python/pyspark/sql/context.py",
> > line 318, in createDataFrame
> >     if has_pandas and isinstance(data, pandas.DataFrame):
> > AttributeError: 'module' object has no attribute 'DataFrame'
> >
> > Would appreciate any pointers.
> >
> > Thanks!
> > Ping
> >
> >
> >
>



-- 
Ping Yan

Re: weird issue with sqlContext.createDataFrame - pyspark 1.3.1

Posted by Davies Liu <da...@databricks.com>.

Is it possible that you have an very old version of pandas, that does
not have DataFrame (or in different submodule).

Could you try this:
```
>>> import pandas
>>> pandas.__version__
'0.14.0'
```

On Thu, Oct 8, 2015 at 10:28 PM, ping yan <sh...@gmail.com> wrote:
> I really cannot figure out what this is about..
> (tried to import pandas, in case that is a dependency, but it didn't help.)
>
>>>> from pyspark.sql import SQLContext
>>>> sqlContext=SQLContext(sc)
>>>> sqlContext.createDataFrame(l).collect()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/python/pyspark/sql/context.py",
> line 318, in createDataFrame
>     if has_pandas and isinstance(data, pandas.DataFrame):
> AttributeError: 'module' object has no attribute 'DataFrame'
>
> Would appreciate any pointers.
>
> Thanks!
> Ping
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org