You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2018/11/16 12:28:19 UTC
Is spark.sql.codegen.factoryMode property really for tests only?
Hi,
While reviewing the changes in 2.4 I stumbled
upon spark.sql.codegen.factoryMode internal configuration property [1]. The
doc says:
> Note that this config works only for tests.
Is that correct? I've got some doubts.
I found that it's used in UnsafeProjection.create [2] (through
CodeGeneratorWithInterpretedFallback.createObject) which is used outside
the tests and so made me think if "this config works only for tests" part
is correct.
Are my doubts correct? If not, what am I missing? Thanks.
[1]
https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767
[2]
https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski
Re: Is spark.sql.codegen.factoryMode property really for tests only?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Marco,
Many thanks for such a quick response. With that, I'll direct my curiosity
into a different direction. Thanks!
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski
On Fri, Nov 16, 2018 at 1:44 PM Marco Gaido <ma...@gmail.com> wrote:
> Hi Jacek,
>
> I do believe it is correct. Please check the method you mentioned
> (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant
> only if Utils.isTesting.
>
> Thanks,
> Marco
>
> Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski <ja...@japila.pl>
> ha scritto:
>
>> Hi,
>>
>> While reviewing the changes in 2.4 I stumbled
>> upon spark.sql.codegen.factoryMode internal configuration property [1]. The
>> doc says:
>>
>> > Note that this config works only for tests.
>>
>> Is that correct? I've got some doubts.
>>
>> I found that it's used in UnsafeProjection.create [2] (through
>> CodeGeneratorWithInterpretedFallback.createObject) which is used outside
>> the tests and so made me think if "this config works only for tests" part
>> is correct.
>>
>> Are my doubts correct? If not, what am I missing? Thanks.
>>
>> [1]
>> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767
>>
>> [2]
>> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> Mastering Spark SQL https://bit.ly/mastering-spark-sql
>> Spark Structured Streaming https://bit.ly/spark-structured-streaming
>> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
>> Follow me at https://twitter.com/jaceklaskowski
>>
>
Re: Is spark.sql.codegen.factoryMode property really for tests only?
Posted by Marco Gaido <ma...@gmail.com>.
Hi Jacek,
I do believe it is correct. Please check the method you mentioned
(CodeGeneratorWithInterpretedFallback.createObject): the value is relevant
only if Utils.isTesting.
Thanks,
Marco
Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski <ja...@japila.pl>
ha scritto:
> Hi,
>
> While reviewing the changes in 2.4 I stumbled
> upon spark.sql.codegen.factoryMode internal configuration property [1]. The
> doc says:
>
> > Note that this config works only for tests.
>
> Is that correct? I've got some doubts.
>
> I found that it's used in UnsafeProjection.create [2] (through
> CodeGeneratorWithInterpretedFallback.createObject) which is used outside
> the tests and so made me think if "this config works only for tests" part
> is correct.
>
> Are my doubts correct? If not, what am I missing? Thanks.
>
> [1]
> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767
>
> [2]
> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> Mastering Spark SQL https://bit.ly/mastering-spark-sql
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
> Follow me at https://twitter.com/jaceklaskowski
>