You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2018/05/30 18:09:29 UTC
[SQL] Purpose of RuntimeReplaceable unevaluable unary expressions?
Hi,
I've been exploring RuntimeReplaceable expressions [1] and have been
wondering what their purpose is.
Quoting the scaladoc [2]:
> An expression that gets replaced at runtime (currently by the optimizer)
into a different expression for evaluation. This is mainly used to provide
compatibility with other databases.
For example, ParseToTimestamp expression is a RuntimeReplaceable expression
and it is replaced by Cast(left, TimestampType) or Cast(UnixTimestamp(left,
format), TimestampType) per to_timestamp function (there are two variants).
My question is why is this RuntimeReplaceable better than simply using the
Casts as the implementation of to_timestamp functions?
def to_timestamp(s: Column, fmt: String): Column = withExpr {
// pseudocode
Cast(UnixTimestamp(left, format), TimestampType)
}
What's wrong with the above implementation compared to the current one?
[1]
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L275
[2]
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L266-L267
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski
Re: [SQL] Purpose of RuntimeReplaceable unevaluable unary expressions?
Posted by Jacek Laskowski <ja...@japila.pl>.
Yay! That's right!!! Thanks Reynold. Such a short answer with so much
information. Thanks.
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski
On Wed, May 30, 2018 at 8:10 PM, Reynold Xin <rx...@databricks.com> wrote:
> SQL expressions?
>
> On Wed, May 30, 2018 at 11:09 AM Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> I've been exploring RuntimeReplaceable expressions [1] and have been
>> wondering what their purpose is.
>>
>> Quoting the scaladoc [2]:
>>
>> > An expression that gets replaced at runtime (currently by the
>> optimizer) into a different expression for evaluation. This is mainly used
>> to provide compatibility with other databases.
>>
>> For example, ParseToTimestamp expression is a RuntimeReplaceable
>> expression and it is replaced by Cast(left, TimestampType)
>> or Cast(UnixTimestamp(left, format), TimestampType) per to_timestamp
>> function (there are two variants).
>>
>> My question is why is this RuntimeReplaceable better than simply using
>> the Casts as the implementation of to_timestamp functions?
>>
>> def to_timestamp(s: Column, fmt: String): Column = withExpr {
>> // pseudocode
>> Cast(UnixTimestamp(left, format), TimestampType)
>> }
>>
>> What's wrong with the above implementation compared to the current one?
>>
>> [1] https://github.com/apache/spark/blob/master/sql/
>> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
>> expressions/Expression.scala#L275
>>
>> [2] https://github.com/apache/spark/blob/master/sql/
>> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
>> expressions/Expression.scala#L266-L267
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> Mastering Spark SQL https://bit.ly/mastering-spark-sql
>> Spark Structured Streaming https://bit.ly/spark-structured-streaming
>> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
>> Follow me at https://twitter.com/jaceklaskowski
>>
>
Re: [SQL] Purpose of RuntimeReplaceable unevaluable unary expressions?
Posted by Reynold Xin <rx...@databricks.com>.
SQL expressions?
On Wed, May 30, 2018 at 11:09 AM Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> I've been exploring RuntimeReplaceable expressions [1] and have been
> wondering what their purpose is.
>
> Quoting the scaladoc [2]:
>
> > An expression that gets replaced at runtime (currently by the optimizer)
> into a different expression for evaluation. This is mainly used to provide
> compatibility with other databases.
>
> For example, ParseToTimestamp expression is a RuntimeReplaceable
> expression and it is replaced by Cast(left, TimestampType)
> or Cast(UnixTimestamp(left, format), TimestampType) per to_timestamp
> function (there are two variants).
>
> My question is why is this RuntimeReplaceable better than simply using the
> Casts as the implementation of to_timestamp functions?
>
> def to_timestamp(s: Column, fmt: String): Column = withExpr {
> // pseudocode
> Cast(UnixTimestamp(left, format), TimestampType)
> }
>
> What's wrong with the above implementation compared to the current one?
>
> [1]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L275
>
> [2]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L266-L267
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> Mastering Spark SQL https://bit.ly/mastering-spark-sql
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
> Follow me at https://twitter.com/jaceklaskowski
>