You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Zakaria Hili <za...@gmail.com> on 2016/10/28 08:26:31 UTC

convert spark dataframe to numpy (ndarray)

Hi,

Is there any way to convert a spark dataframe into numpy ndarray without
using toPandas operation ?

Example:

C1   C2  C3   C4 0.7 3.0 1000 109540.9 4.2 1200 12345

I want to get this output:

[(0.7, 3.0, 1000L, 10954),(0.9, 4.2, 1200L, 12345)],
    dtype=[('C1', '<f8'), ('C2', '<f8'), ('C3', '<i8'), ('C4', '<f8'),(2,))])

NB:I'm working with pyspark 1.6.2


Thank you
ᐧ