You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/09/23 15:28:45 UTC
Why Expression.deterministic method and Nondeterministic trait?
Hi,
Just came across the Expression trait [1] that can be check for
determinism by the method deterministic [2] and trait Nondeterministic
[3]. Why both?
[1] https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L53
[2] https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L80
[3] https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L271
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Why Expression.deterministic method and Nondeterministic trait?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Reynold,
That's my point indeed. If non-determinism is a trait/property of a
class (objects really at runtime) it's by definition part of a trait
and only Nondeterministic expressions are...well...non-deterministic.
I'm yet to review the code regarding the trait, but I suspect that
there are places where you check whether an expression is
deterministic by the method not the trait. If so, there's this
ambiguity I'm talking about.
Anyway, I'm glad to learn from you guys! Thanks.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sat, Sep 24, 2016 at 12:35 AM, Reynold Xin <rx...@databricks.com> wrote:
> deterministic method describes whether this instance of the expression tree
> is deterministic, whereas Nondeterministic trait is about a class.
>
>
> On Fri, Sep 23, 2016 at 10:46 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi Herman,
>>
>> That helps to know that someone can explain why we've got the two
>> nondeterministic states.
>>
>> It's not possible to say...a non-Nondeterministic expression can be
>> non-deterministic (the former is the trait while the latter is the
>> method) #strange
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Fri, Sep 23, 2016 at 6:44 PM, Herman van Hövell tot Westerflier
>> <hv...@databricks.com> wrote:
>> > Jacek,
>> >
>> > A non-deterministic expression usually holds some state. The
>> > Nondeterministic trait makes sure a user can initialize this state
>> > properly.
>> > Take a look at InterpretedProjection for instance.
>> >
>> > HTH
>> >
>> > -Herman
>> >
>> > On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski <ja...@japila.pl>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> Just came across the Expression trait [1] that can be check for
>> >> determinism by the method deterministic [2] and trait Nondeterministic
>> >> [3]. Why both?
>> >>
>> >> [1]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L53
>> >> [2]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L80
>> >> [3]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L271
>> >>
>> >> Pozdrawiam,
>> >> Jacek Laskowski
>> >> ----
>> >> https://medium.com/@jaceklaskowski/
>> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> >> Follow me at https://twitter.com/jaceklaskowski
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Why Expression.deterministic method and Nondeterministic trait?
Posted by Reynold Xin <rx...@databricks.com>.
deterministic method describes whether this instance of the expression tree
is deterministic, whereas Nondeterministic trait is about a class.
On Fri, Sep 23, 2016 at 10:46 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi Herman,
>
> That helps to know that someone can explain why we've got the two
> nondeterministic states.
>
> It's not possible to say...a non-Nondeterministic expression can be
> non-deterministic (the former is the trait while the latter is the
> method) #strange
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Fri, Sep 23, 2016 at 6:44 PM, Herman van Hövell tot Westerflier
> <hv...@databricks.com> wrote:
> > Jacek,
> >
> > A non-deterministic expression usually holds some state. The
> > Nondeterministic trait makes sure a user can initialize this state
> properly.
> > Take a look at InterpretedProjection for instance.
> >
> > HTH
> >
> > -Herman
> >
> > On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski <ja...@japila.pl>
> wrote:
> >>
> >> Hi,
> >>
> >> Just came across the Expression trait [1] that can be check for
> >> determinism by the method deterministic [2] and trait Nondeterministic
> >> [3]. Why both?
> >>
> >> [1]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L53
> >> [2]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L80
> >> [3]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L271
> >>
> >> Pozdrawiam,
> >> Jacek Laskowski
> >> ----
> >> https://medium.com/@jaceklaskowski/
> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> >> Follow me at https://twitter.com/jaceklaskowski
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Why Expression.deterministic method and Nondeterministic trait?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Herman,
That helps to know that someone can explain why we've got the two
nondeterministic states.
It's not possible to say...a non-Nondeterministic expression can be
non-deterministic (the former is the trait while the latter is the
method) #strange
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Fri, Sep 23, 2016 at 6:44 PM, Herman van Hövell tot Westerflier
<hv...@databricks.com> wrote:
> Jacek,
>
> A non-deterministic expression usually holds some state. The
> Nondeterministic trait makes sure a user can initialize this state properly.
> Take a look at InterpretedProjection for instance.
>
> HTH
>
> -Herman
>
> On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi,
>>
>> Just came across the Expression trait [1] that can be check for
>> determinism by the method deterministic [2] and trait Nondeterministic
>> [3]. Why both?
>>
>> [1]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L53
>> [2]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L80
>> [3]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L271
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Why Expression.deterministic method and Nondeterministic trait?
Posted by Herman van Hövell tot Westerflier <hv...@databricks.com>.
Jacek,
A non-deterministic expression usually holds some state. The
Nondeterministic trait makes sure a user can initialize this state
properly. Take a look at InterpretedProjection
<https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L66-L69>
for instance.
HTH
-Herman
On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Just came across the Expression trait [1] that can be check for
> determinism by the method deterministic [2] and trait Nondeterministic
> [3]. Why both?
>
> [1] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L53
> [2] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L80
> [3] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L271
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>