You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gaspar Muñoz <gm...@datiobd.com> on 2022/01/20 07:55:04 UTC
Code fails when AQE enabled in Spark 3.1
Hi guys,
hundreds of spark jobs run on my company every day. We are running Spark
3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them.
We can't upgrade to 3.2 right now so we want enable it explicitly using
appropriate conf when spark submit.
Some of them fails when enable AQE but I can't discover what is happening.
In order to give your information I prepared a small snippet for spark
shell that fails in Spark 3.1 when AQE enabled and works when disabled. It
also work in 3.2 but I think maybe is a bug that can be fixed for 3.1.3.
The code and explanation can be found here:
https://issues.apache.org/jira/browse/SPARK-37898
Regards
--
Gaspar Muñoz Soria
Re: Code fails when AQE enabled in Spark 3.1
Posted by Sean Owen <sr...@gmail.com>.
At a glance, it doesn't seem so. That is a corner case in two ways - very
old dates and using RDDs, at least it seems.
I also suspect that individual change is tied to a lot of other date
related changes in 3.2, so may not be very back-portable.
You should pursue updating to 3.2 for many reasons, but this too if it
affects you.
On Tue, Feb 1, 2022 at 1:50 AM Gaspar Muñoz <gm...@datiobd.com> wrote:
> it looks that this commit (
> https://github.com/apache/spark/commit/a85490659f45410be3588c669248dc4f534d2a71)
> do the trick.
>
> [image: image.png]
>
> Don't you think, this bug is enough important to incluide in 3.1 branch?
>
> Regards
>
> El jue, 20 ene 2022 a las 8:55, Gaspar Muñoz (<gm...@datiobd.com>)
> escribió:
>
>> Hi guys,
>>
>> hundreds of spark jobs run on my company every day. We are running Spark
>> 3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them.
>> We can't upgrade to 3.2 right now so we want enable it explicitly using
>> appropriate conf when spark submit.
>>
>> Some of them fails when enable AQE but I can't discover what is
>> happening. In order to give your information I prepared a small snippet
>> for spark shell that fails in Spark 3.1 when AQE enabled and works when
>> disabled. It also work in 3.2 but I think maybe is a bug that can be fixed
>> for 3.1.3.
>>
>> The code and explanation can be found here:
>> https://issues.apache.org/jira/browse/SPARK-37898
>>
>> Regards
>> --
>> Gaspar Muñoz Soria
>>
>
>
> --
> Gaspar Muñoz Soria
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 828 6473
>
Re: Code fails when AQE enabled in Spark 3.1
Posted by Gaspar Muñoz <gm...@datiobd.com>.
it looks that this commit (
https://github.com/apache/spark/commit/a85490659f45410be3588c669248dc4f534d2a71)
do the trick.
[image: image.png]
Don't you think, this bug is enough important to incluide in 3.1 branch?
Regards
El jue, 20 ene 2022 a las 8:55, Gaspar Muñoz (<gm...@datiobd.com>)
escribió:
> Hi guys,
>
> hundreds of spark jobs run on my company every day. We are running Spark
> 3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them.
> We can't upgrade to 3.2 right now so we want enable it explicitly using
> appropriate conf when spark submit.
>
> Some of them fails when enable AQE but I can't discover what is
> happening. In order to give your information I prepared a small snippet
> for spark shell that fails in Spark 3.1 when AQE enabled and works when
> disabled. It also work in 3.2 but I think maybe is a bug that can be fixed
> for 3.1.3.
>
> The code and explanation can be found here:
> https://issues.apache.org/jira/browse/SPARK-37898
>
> Regards
> --
> Gaspar Muñoz Soria
>
--
Gaspar Muñoz Soria
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473