You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "eaton (JIRA)" <ji...@apache.org> on 2017/08/22 10:26:00 UTC
[jira] [Updated] (SPARK-21807) The getAliasedConstraints function
in LogicalPlan will take a long time when number of expressions is greater
than 100
[ https://issues.apache.org/jira/browse/SPARK-21807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
eaton updated SPARK-21807:
--------------------------
Description:
The getAliasedConstraints fuction in LogicalPlan.scala will clone the expression set when an element added,
and it will take a long time.
Before modified, the cost of getAliasedConstraints is:
100 expressions: 41 seconds
150 expressions: 466 seconds
The test is like this:
test("getAliasedConstraints") {
val expressionNum = 150
val aggExpression = (1 to expressionNum).map(i => Alias(Count(Literal(1)), s"cnt$i")())
val aggPlan = Aggregate(Nil, aggExpression, LocalRelation())
val beginTime = System.currentTimeMillis()
val expressions = aggPlan.validConstraints
println(s"validConstraints cost: ${System.currentTimeMillis() - beginTime}ms")
// The size of Aliased expression is n * (n - 1) / 2 + n
assert( expressions.size === expressionNum * (expressionNum - 1) / 2 + expressionNum)
}
was:
The getAliasedConstraints fuction in LogicalPlan.scala will clone the expression set when an element added,
and it will take a long time.
Before modified, the cost of getAliasedConstraints is:
100 expressions: 41 seconds
150 expressions: 466 seconds
> The getAliasedConstraints function in LogicalPlan will take a long time when number of expressions is greater than 100
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-21807
> URL: https://issues.apache.org/jira/browse/SPARK-21807
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: eaton
>
> The getAliasedConstraints fuction in LogicalPlan.scala will clone the expression set when an element added,
> and it will take a long time.
> Before modified, the cost of getAliasedConstraints is:
> 100 expressions: 41 seconds
> 150 expressions: 466 seconds
> The test is like this:
> test("getAliasedConstraints") {
> val expressionNum = 150
> val aggExpression = (1 to expressionNum).map(i => Alias(Count(Literal(1)), s"cnt$i")())
> val aggPlan = Aggregate(Nil, aggExpression, LocalRelation())
> val beginTime = System.currentTimeMillis()
> val expressions = aggPlan.validConstraints
> println(s"validConstraints cost: ${System.currentTimeMillis() - beginTime}ms")
> // The size of Aliased expression is n * (n - 1) / 2 + n
> assert( expressions.size === expressionNum * (expressionNum - 1) / 2 + expressionNum)
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org