You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "balaji krishnan (JIRA)" <ji...@apache.org> on 2017/04/12 12:06:41 UTC
[jira] [Created] (SPARK-20309) Repartitioning - more than the
default number of partitions
balaji krishnan created SPARK-20309:
---------------------------------------
Summary: Repartitioning - more than the default number of partitions
Key: SPARK-20309
URL: https://issues.apache.org/jira/browse/SPARK-20309
Project: Spark
Issue Type: Bug
Components: Java API, SQL
Affects Versions: 1.6.1
Environment: Spark 1.6.1, Hadoop 2.6.2, Redhat Linux
Reporter: balaji krishnan
We have a requirement to process roughly 45MM rows. Each row has a KEY column. The unique number of KEYS are roughly 25000. I do the following Spark SQL to get all 25000 partitions
hiveContext.sql("select * from BigTbl distribute by KEY")
This nicely creates all required partitions. Using MapPartitions we process all of these partitions in parallel. I have a variable which should be initiated once per partition. Lets call that variable name as designName. This variable is initialized with the KEY value (partition value) for future identification purposes.
FlatMapFunction<java.util.Iterator<Row>, DSGType> flatMapSetup = new FlatMapFunction<java.util.Iterator<Row>, DSGType>() {
String designName= null;
@Override
public java.lang.Iterable<DSGType> call(java.util.Iterator<Row> it) throws Exception {
while (it.hasNext()) {
Row row = it.next(); //row is of type org.apache.spark.sql.Row
if (designName == null) {
designName = row.getString(1); // row is of type org.apache.spark.sql.Row
}
.....
}
List<DSGType> Dsg = partitionedRecs.mapPartitions(flatMapSetup).collect();
The problem starts here for me.
Since i have more than the default number of partitions (200) i do a re-partition to the unique number of partitions in my BigTbl. However if i re-partition the designName gets completely confused. Instead of processing a total of 25000 unique values, i get only 15000 values processed. For some reason the repartition completely messes up the uniqueness of the previous step. The statement that i use to 'distribute by' and subsequently repartition
long distinctValues = hiveContext.sql("select KEY from BigTbl").distinct().count();
JavaRDD<Row> partitionedRecs = hiveContext.sql("select * from BigTbl DISTRIBUTE by SKU ").repartition((int)distinctValues).toJavaRDD().cache();
I tried another solution, that is by changing the spark.sql.shuffle.partitions during SparkContext creation. Even that did not help. I get the same issue.
new JavaSparkContext(new SparkConf().set("spark.driver.maxResultSize","0").set("spark.sql.shuffle.partitions","26000"))
Is there a way to solve this issue please or is this a DISTRIBUTE BY bug ? We are using Spark 1.6.1
Regards
Bala
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org