Spark data frame map problem

[Shashank Mandil <ma...@gmail.com>]

Hi All,

I have a spark data frame which has 992 rows inside it.
When I run a map on this data frame I expect that the map should work for
all the 992 rows.

As a mapper runs on an executor on  a cluster I did a distributed count of
the number of rows the mapper is being run on.

dataframe.map(r => {
   //distributed count inside here using zookeeper
})

I have found that this distributed count inside the mapper is not exactly
992. I have found this number to vary with different runs.

Does anybody have any idea what might be happening ? By the way, I am using
spark 1.6.1

Thanks,
Shashank

Re: Spark data frame map problem

["颜发才 (Yan Facai)" <fa...@gmail.com>]

Could you give more details of your code?



On Wed, Mar 22, 2017 at 2:40 AM, Shashank Mandil <ma...@gmail.com>
wrote:

> Hi All,
>
> I have a spark data frame which has 992 rows inside it.
> When I run a map on this data frame I expect that the map should work for
> all the 992 rows.
>
> As a mapper runs on an executor on  a cluster I did a distributed count of
> the number of rows the mapper is being run on.
>
> dataframe.map(r => {
>    //distributed count inside here using zookeeper
> })
>
> I have found that this distributed count inside the mapper is not exactly
> 992. I have found this number to vary with different runs.
>
> Does anybody have any idea what might be happening ? By the way, I am
> using spark 1.6.1
>
> Thanks,
> Shashank
>
>