Unexpected traffic size between Driver and Worker node ?

[Aliaksei Tsyvunchyk <at...@exadel.com>]

Hello guys,

I’ve started learning Spark and during experiments I’ve discovered that Driver and Worker node exchange with big amount of data (Worker send to Driver), and I cannot figure out whether it is correct or I have misconfigured something.
So I have created very simple test case: I’ve decided to sum integer value for some record type. I have imported my test data in parquet file put it to hdfs and start making tests. This test is running on one physical computer.
1. I have configure spark master and slave (run it in standalone mode).
2. I have created simple program which calculates sum.
3. When I run test - I see ~10mb of data transferred from Worker node to driver node. 

Here is code I’m executing:

DataFrame df = sqlContext.sql("Select viewingWeight from EnhancedProgramViewingRecords epvr”);
Number res = df
        .toJavaRDD()
        .map(r->r.getInt(0))
        .reduce((x, y)->x+y);
System.out.println("Result is: " + res);

In spark UI I see that total job input size is 80 Mb (see attached image), and I use nettop macOS utility to monitor socket traffic and see 10mb per job of traffic here(see second attached image).
I expect to have big traffic between hdfs and Worker and some traffic between driver and worker but not 10mb per request. Having this I have a strong feeling that part of work is performing by driver program rather than worker.
What do you think on this ?


-- 


CONFIDENTIALITY NOTICE: This email and files attached to it are 
confidential. If you are not the intended recipient you are hereby notified 
that using, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited. If you have received 
this email in error please notify the sender and delete this email.