You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "sunerhan1992@sina.com" <su...@sina.com> on 2017/06/20 12:10:35 UTC
dataframe mappartitions problem
hello，
I'm using dataframe.mappartitions(r=>myfunc(r)) and it works so slow if if i do nothing in my func but return an empty iterator
Following are two dags about not using and using mappartitions and it,and using mappartitions costs more time even if myfunc do nothing.
Is there a problem about scheduling stages when mappartitions involved?
sunerhan1992@sina.com
Re: Re: dataframe mappartitions problem
Posted by "sunerhan1992@sina.com" <su...@sina.com>.
hello，
I repost dag picture in appendix,please check.
BTW,i'm using spark1.6.3 and try dataframe generated from hivecontext
sunerhan1992@sina.com
From: Wenchen Fan
Date: 2017-06-20 20:42
To: sunerhan1992@sina.com
CC: dev@spark.apache.org
Subject: Re: dataframe mappartitions problem
`Dataset.mapPartitions` takes `func: Iterator[T] => Iterator[U]`, which means, spark need to deserialize the internal binary format to type `T`, and this deserialization is costly.
If you do need to do some hack, you can use the internal API: `Dataset.queryExecution.toRdd.mapPartitions`, which has no compatibility guarantees, and you need to deal with `InternalRow` directly.
On 20 Jun 2017, at 8:10 PM, sunerhan1992@sina.com wrote:
hello，
I'm using dataframe.mappartitions(r=>myfunc(r)) and it works so slow if if i do nothing in my func but return an empty iterator
Following are two dags about not using and using mappartitions and it,and using mappartitions costs more time even if myfunc do nothing.
Is there a problem about scheduling stages when mappartitions involved?
<Catch.jpg>
<Catch696F.jpg>
sunerhan1992@sina.com
Re: dataframe mappartitions problem
Posted by Wenchen Fan <cl...@gmail.com>.
`Dataset.mapPartitions` takes `func: Iterator[T] => Iterator[U]`, which means, spark need to deserialize the internal binary format to type `T`, and this deserialization is costly.
If you do need to do some hack, you can use the internal API: `Dataset.queryExecution.toRdd.mapPartitions`, which has no compatibility guarantees, and you need to deal with `InternalRow` directly.
> On 20 Jun 2017, at 8:10 PM, sunerhan1992@sina.com wrote:
>
> hello，
> I'm using dataframe.mappartitions(r=>myfunc(r)) and it works so slow if if i do nothing in my func but return an empty iterator
> Following are two dags about not using and using mappartitions and it,and using mappartitions costs more time even if myfunc do nothing.
> Is there a problem about scheduling stages when mappartitions involved?
> <Catch.jpg>
> <Catch696F.jpg>
> sunerhan1992@sina.com <ma...@sina.com>