You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "caoxuewen (JIRA)" <ji...@apache.org> on 2017/09/16 10:52:00 UTC
[jira] [Updated] (SPARK-22035) the value of statistical
logicalPlan.stats.sizeInBytes which is not expected
[ https://issues.apache.org/jira/browse/SPARK-22035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
caoxuewen updated SPARK-22035:
------------------------------
Summary: the value of statistical logicalPlan.stats.sizeInBytes which is not expected (was: Improving the value of statistical logicalPlan.stats.sizeInBytes which is not expected)
> the value of statistical logicalPlan.stats.sizeInBytes which is not expected
> ----------------------------------------------------------------------------
>
> Key: SPARK-22035
> URL: https://issues.apache.org/jira/browse/SPARK-22035
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: caoxuewen
>
> Currently, assume there will be the same number of rows as child has. statistics `logicalPlan.stats.SizeInBytes` is calculated based on the percentage of the size of the child's data type and the size of the current data type size. But there is a problem. Statistics are not very accurate. for example:
> ```
> val N = 1 << 3
> val df = spark.range(N).selectExpr("id as k1",
> s"cast(id % 3 as string) as idString1",
> s"cast((id + 1) % 5 as string) as idString3")
> val sizeInBytes = df.logicalPlan.stats.sizeInBytes
> println("sizeInBytes : " + sizeInBytes)
> ```
> before modify：
> sizeInBytes is 224(8 * 8 * ( (8 + 20 + 20 +8) / (8 + 8))).
> debug information in ` SizeInBytesOnlyStatsPlanVisitor.visitUnaryNode `
> ```
> p.child.dataType: LongType defaultSize: 8
> p.dataType: LongType defaultSize: 8
> p.dataType: StringType defaultSize: 20
> p.dataType: StringType defaultSize: 20
> childRowSize: 16 outputRowSize: 56
> p.child.stats.sizeInBytes : 64
> p.stats.sizeInBytes : 224
> sizeInBytes: 224
> ```
> but sizeInBytes must be 384( 8 * (8 + 20 + 20) ).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org