You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/12/05 20:37:00 UTC
[jira] [Updated] (SPARK-50502) Support pushing DeserializeToObject through Union
[ https://issues.apache.org/jira/browse/SPARK-50502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-50502:
-----------------------------------
Labels: pull-request-available (was: )
> Support pushing DeserializeToObject through Union
> -------------------------------------------------
>
> Key: SPARK-50502
> URL: https://issues.apache.org/jira/browse/SPARK-50502
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Emil Ejbyfeldt
> Priority: Major
> Labels: pull-request-available
>
> When migrating some RDD code to Dataset we noticed a significant performance degradation. The degradation was caused by an extra serialize/deserialize round trip.
> Consider the following plan
> {code:java}
> val ds1 = spark.createDataset(Seq((1,2)))
> val ds2 = sc.parallelize(Seq((1,2))).toDS().filter(_._1 > 1)
> ds1.union(ds2).map(identity).explain(){code}
> {code:java}
> == Physical Plan ==
> *(2) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#353, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#354]
> +- *(2) MapElements $line43.$read$$iw$$Lambda$4181/0x00007f3d4d10ae10@24287e1f, obj#352: scala.Tuple2
> +- *(2) DeserializeToObject newInstance(class scala.Tuple2), obj#351: scala.Tuple2
> +- Union
> :- LocalTableScan [_1#318, _2#319]
> +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#326, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#327]
> +- *(1) Filter $line40.$read$$iw$$Lambda$4179/0x00007f3d4d10ecc8@d06af57.apply
> +- Scan[obj#325]{code}
> if we pushed the DeserializeToObject through the Union we be able to cancel it against the SerializeFromObject and we could perform the union on the deserialized objects instead.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org