You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "deszuc@163.com" <de...@163.com> on 2018/06/25 14:15:05 UTC

[Spark SQL] was it correct that only one executor was used to shuffle the data for reduce task?

hi,all
    i'm using spark-2.0.0 on hdp 2.5.0 to build a spark-sql app,below is the spark-submit configuration:
spark-submit    \
--class "FASTMDTFlow"     \
--master yarn \
--deploy-mode client \
--driver-memory 12g \
--num-executors 110    \
--executor-memory 8g \
--executor-cores 3 \
--conf "spark.driver.maxResultSize=2g" \
--conf "spark.sql.codegen=true" \
--conf "spark.default.parallelism=360" \
--conf "spark.sql.shuffle.partitions=800" \
my.jar

The app was written in  scala with pure sql statements(simply contained the sql statements within scala statements),
The Sql statements  are composed of some 'avg,sum'  aggregated functions and 'group by' clause.
when the app was  running, In the  'Aggregated Metrics by Executor' on the spark UI table ,only one executor had 'Shuffle Read Size / Records' value (about 100GB),others were zero.
i saw it as there was only one reduce(shuffle) task for reading data and writing data.
 was  this  correct?
(if the information was not enough,please tell me.)




By Luo Hai 
Best Wishes!
deszuc@163.com