You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel de Oliveira Mantovani <da...@gmail.com> on 2021/12/13 15:40:48 UTC
spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Hello team,
I've found this issue while I was porting my project from Apache Spark
3.1.x to 3.2.x.
https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
Do we have a bug for that in apache-spark or I need to create one ?
Thank you so much
[info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
[info] org.apache.spark.sql.AnalysisException: Recursive view `__TABLE__`
detected (cycle: `__TABLE__` -> `__TABLE__`)
[info] at
org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
[info] at
org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
[info] at
org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
[info] at
org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
[info] at scala.collection.Iterator.foreach(Iterator.scala:941)
[info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
[info] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
[info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[info] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
I am pretty sure that Sean already answered the question. Also i do not
think that creating iterative table definitions or data frame definitions
are best practice.
Regards,
Gourav
On Mon, Dec 13, 2021 at 4:00 PM Sean Owen <sr...@gmail.com> wrote:
> ... but the error is not "because that already exists". See your stack
> trace. It's because the definition is recursive. You define temp view
> test1, create a second DF from it, and then redefine test1 as that result.
> test1 depends on test1.
>
> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> Sean,
>>
>> The method name is very clear "createOrReplaceTempView" doesn't make any
>> sense to throw an exception because this view already exists. Spark 3.2.x
>> is breaking back compatibility with no reason or sense.
>>
>>
>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> The error looks 'valid' - you define a temp view in terms of its own
>>> previous version, which doesn't quite make sense - somewhere the new
>>> definition depends on the old definition. I think it just correctly
>>> surfaces as an error now,.
>>>
>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>
>>>> Hello team,
>>>>
>>>> I've found this issue while I was porting my project from Apache Spark
>>>> 3.1.x to 3.2.x.
>>>>
>>>>
>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>
>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>
>>>> Thank you so much
>>>>
>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>> [info] at
>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>> [info] at
>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>> [info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>> [info] at
>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>> [info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>
>>>> --
>>>>
>>>> --
>>>> Daniel Mantovani
>>>>
>>>>
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
You are correct, I understand. My only concern is the back compatibility
problem, which worked for the previous version of Apache Spark. It's
painful when an OOTB feature breaks without documentation or a workaround
like "spark.sql.legacy.keepSqlRecursive" true/false. It's not about "my
code", it is about all production code running out there.
Thank you so much
On Mon, Dec 13, 2021 at 2:32 PM Sean Owen <sr...@gmail.com> wrote:
> I think we're around in circles - you should not do this. You essentially
> have "__TABLE__ = SELECT * FROM __TABLE__" and I hope it's clear why that
> can't work in general.
> At first execution, sure, maybe "old" __TABLE__ refers to "SELECT 1", but
> what about the second time? if you stick to that interpretation, it's
> actually not executing correctly, though 'works'. If you execute it as is,
> it fails for circularity. Both are bad, so it's just disallowed.
> Just fix your code?
>
> On Mon, Dec 13, 2021 at 11:27 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> I've reduced the code to reproduce the issue,
>>
>> val df = spark.sql("SELECT 1")
>> df.createOrReplaceTempView("__TABLE__")
>> spark.sql("SELECT * FROM __TABLE__").show
>> val df2 = spark.sql("SELECT *,2 FROM __TABLE__")
>> df2.createOrReplaceTempView("__TABLE__") // Exception in Spark 3.2 but
>> works for Spark 2.4.x and Spark 3.1.x
>> spark.sql("SELECT * FROM __TABLE__").show
>>
>> org.apache.spark.sql.AnalysisException: Recursive view `__TABLE__`
>> detected (cycle: `__TABLE__` -> `__TABLE__`)
>> at
>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>> at
>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>> at
>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>> at
>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>
>> On Mon, Dec 13, 2021 at 2:10 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> _shrug_ I think this is a bug fix, unless I am missing something here.
>>> You shouldn't just use __TABLE__ for everything, and I'm not seeing a good
>>> reason to do that other than it's what you do now.
>>> I'm not clear if it's coming across that this _can't_ work in the
>>> general case.
>>>
>>> On Mon, Dec 13, 2021 at 11:03 AM Daniel de Oliveira Mantovani <
>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>
>>>>
>>>> In this context, I don't want to worry about the name of the temporary
>>>> table. That's why it is "__TABLE__".
>>>> The point is that this behavior for Spark 3.2.x it's breaking back
>>>> compatibility for all previous versions of Apache Spark. In my opinion we
>>>> should at least have some flag like "spark.sql.legacy.keepSqlRecursive"
>>>> true/false.
>>>>
>>>
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
I think we're around in circles - you should not do this. You essentially
have "__TABLE__ = SELECT * FROM __TABLE__" and I hope it's clear why that
can't work in general.
At first execution, sure, maybe "old" __TABLE__ refers to "SELECT 1", but
what about the second time? if you stick to that interpretation, it's
actually not executing correctly, though 'works'. If you execute it as is,
it fails for circularity. Both are bad, so it's just disallowed.
Just fix your code?
On Mon, Dec 13, 2021 at 11:27 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
> I've reduced the code to reproduce the issue,
>
> val df = spark.sql("SELECT 1")
> df.createOrReplaceTempView("__TABLE__")
> spark.sql("SELECT * FROM __TABLE__").show
> val df2 = spark.sql("SELECT *,2 FROM __TABLE__")
> df2.createOrReplaceTempView("__TABLE__") // Exception in Spark 3.2 but
> works for Spark 2.4.x and Spark 3.1.x
> spark.sql("SELECT * FROM __TABLE__").show
>
> org.apache.spark.sql.AnalysisException: Recursive view `__TABLE__`
> detected (cycle: `__TABLE__` -> `__TABLE__`)
> at
> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
> at
> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
> at
> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
> at
> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>
> On Mon, Dec 13, 2021 at 2:10 PM Sean Owen <sr...@gmail.com> wrote:
>
>> _shrug_ I think this is a bug fix, unless I am missing something here.
>> You shouldn't just use __TABLE__ for everything, and I'm not seeing a good
>> reason to do that other than it's what you do now.
>> I'm not clear if it's coming across that this _can't_ work in the general
>> case.
>>
>> On Mon, Dec 13, 2021 at 11:03 AM Daniel de Oliveira Mantovani <
>> daniel.oliveira.mantovani@gmail.com> wrote:
>>
>>>
>>> In this context, I don't want to worry about the name of the temporary
>>> table. That's why it is "__TABLE__".
>>> The point is that this behavior for Spark 3.2.x it's breaking back
>>> compatibility for all previous versions of Apache Spark. In my opinion we
>>> should at least have some flag like "spark.sql.legacy.keepSqlRecursive"
>>> true/false.
>>>
>>
>
> --
>
> --
> Daniel Mantovani
>
>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
I've reduced the code to reproduce the issue,
val df = spark.sql("SELECT 1")
df.createOrReplaceTempView("__TABLE__")
spark.sql("SELECT * FROM __TABLE__").show
val df2 = spark.sql("SELECT *,2 FROM __TABLE__")
df2.createOrReplaceTempView("__TABLE__") // Exception in Spark 3.2 but
works for Spark 2.4.x and Spark 3.1.x
spark.sql("SELECT * FROM __TABLE__").show
org.apache.spark.sql.AnalysisException: Recursive view `__TABLE__` detected
(cycle: `__TABLE__` -> `__TABLE__`)
at
org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
at
org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
at
org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
at
org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
On Mon, Dec 13, 2021 at 2:10 PM Sean Owen <sr...@gmail.com> wrote:
> _shrug_ I think this is a bug fix, unless I am missing something here. You
> shouldn't just use __TABLE__ for everything, and I'm not seeing a good
> reason to do that other than it's what you do now.
> I'm not clear if it's coming across that this _can't_ work in the general
> case.
>
> On Mon, Dec 13, 2021 at 11:03 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>>
>> In this context, I don't want to worry about the name of the temporary
>> table. That's why it is "__TABLE__".
>> The point is that this behavior for Spark 3.2.x it's breaking back
>> compatibility for all previous versions of Apache Spark. In my opinion we
>> should at least have some flag like "spark.sql.legacy.keepSqlRecursive"
>> true/false.
>>
>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
_shrug_ I think this is a bug fix, unless I am missing something here. You
shouldn't just use __TABLE__ for everything, and I'm not seeing a good
reason to do that other than it's what you do now.
I'm not clear if it's coming across that this _can't_ work in the general
case.
On Mon, Dec 13, 2021 at 11:03 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
>
> In this context, I don't want to worry about the name of the temporary
> table. That's why it is "__TABLE__".
> The point is that this behavior for Spark 3.2.x it's breaking back
> compatibility for all previous versions of Apache Spark. In my opinion we
> should at least have some flag like "spark.sql.legacy.keepSqlRecursive"
> true/false.
>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
In this context, I don't want to worry about the name of the temporary
table. That's why it is "__TABLE__".
The point is that this behavior for Spark 3.2.x it's breaking back
compatibility for all previous versions of Apache Spark. In my opinion we
should at least have some flag like "spark.sql.legacy.keepSqlRecursive"
true/false.
On Mon, Dec 13, 2021 at 1:47 PM Sean Owen <sr...@gmail.com> wrote:
> You can replace temp views. Again: what you can't do here is define a temp
> view in terms of itself. If you are reusing the same name over and over,
> it's probably easy to do that, so you don't want to do that. You want
> different names for different temp views, or else ensure you aren't doing
> the kind of thing shown in the SO post. You get the problem right?
>
> On Mon, Dec 13, 2021 at 10:43 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> I didn't post the SO issue, I've just found the same exception I'm facing
>> for Spark 3.2. Almaren Framework has a concept of create temporary views
>> with the name "__TABLE__".
>>
>> Example, if you want to use SQL dialect to a DataFrame to join a
>> table/aggregation/apply a function whatever. Instead of you create a
>> temporary table you just use the "__TABLE__" alias. You don't really care
>> about the name of the table. You may use this "__TABLE__" approach in
>> different parts of your code.
>>
>> Why can't I create or replace temporary views in different DataFrame with
>> the same name as before ?
>>
>>
>>
>> On Mon, Dec 13, 2021 at 1:27 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> If the issue is what you posted in SO, I think the stack trace explains
>>> it already. You want to avoid this recursive definition, which in general
>>> can't work.
>>> I think it's simply explicitly disallowed in all cases now, but, you
>>> should not be depending on this anyway - why can't this just be avoided?
>>>
>>> On Mon, Dec 13, 2021 at 10:06 AM Daniel de Oliveira Mantovani <
>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>
>>>> Sean,
>>>>
>>>> https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2
>>>>
>>>> Just executing "sbt test" will reproduce the error, the same code works
>>>> for spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ?
>>>>
>>>> Thank you so much
>>>>
>>>>
>>>>
>>>> On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> ... but the error is not "because that already exists". See your stack
>>>>> trace. It's because the definition is recursive. You define temp view
>>>>> test1, create a second DF from it, and then redefine test1 as that result.
>>>>> test1 depends on test1.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
>>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>>
>>>>>> Sean,
>>>>>>
>>>>>> The method name is very clear "createOrReplaceTempView" doesn't make
>>>>>> any sense to throw an exception because this view already exists. Spark
>>>>>> 3.2.x is breaking back compatibility with no reason or sense.
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>>>
>>>>>>> The error looks 'valid' - you define a temp view in terms of its own
>>>>>>> previous version, which doesn't quite make sense - somewhere the new
>>>>>>> definition depends on the old definition. I think it just correctly
>>>>>>> surfaces as an error now,.
>>>>>>>
>>>>>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>>>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello team,
>>>>>>>>
>>>>>>>> I've found this issue while I was porting my project from Apache
>>>>>>>> Spark 3.1.x to 3.2.x.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>>>>>
>>>>>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>>>>>
>>>>>>>> Thank you so much
>>>>>>>>
>>>>>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>>>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>>>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>>>>>> [info] at
>>>>>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>>>>>> [info] at
>>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>>>>>> [info] at
>>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>>>>>> [info] at
>>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>>>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>>>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>>>>>> [info] at
>>>>>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>>>>>> [info] at
>>>>>>>> scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>>>>>> [info] at
>>>>>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>>>>>> [info] at
>>>>>>>> scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel Mantovani
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> --
>>>>>> Daniel Mantovani
>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> --
>>>> Daniel Mantovani
>>>>
>>>>
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
You can replace temp views. Again: what you can't do here is define a temp
view in terms of itself. If you are reusing the same name over and over,
it's probably easy to do that, so you don't want to do that. You want
different names for different temp views, or else ensure you aren't doing
the kind of thing shown in the SO post. You get the problem right?
On Mon, Dec 13, 2021 at 10:43 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
> I didn't post the SO issue, I've just found the same exception I'm facing
> for Spark 3.2. Almaren Framework has a concept of create temporary views
> with the name "__TABLE__".
>
> Example, if you want to use SQL dialect to a DataFrame to join a
> table/aggregation/apply a function whatever. Instead of you create a
> temporary table you just use the "__TABLE__" alias. You don't really care
> about the name of the table. You may use this "__TABLE__" approach in
> different parts of your code.
>
> Why can't I create or replace temporary views in different DataFrame with
> the same name as before ?
>
>
>
> On Mon, Dec 13, 2021 at 1:27 PM Sean Owen <sr...@gmail.com> wrote:
>
>> If the issue is what you posted in SO, I think the stack trace explains
>> it already. You want to avoid this recursive definition, which in general
>> can't work.
>> I think it's simply explicitly disallowed in all cases now, but, you
>> should not be depending on this anyway - why can't this just be avoided?
>>
>> On Mon, Dec 13, 2021 at 10:06 AM Daniel de Oliveira Mantovani <
>> daniel.oliveira.mantovani@gmail.com> wrote:
>>
>>> Sean,
>>>
>>> https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2
>>>
>>> Just executing "sbt test" will reproduce the error, the same code works
>>> for spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ?
>>>
>>> Thank you so much
>>>
>>>
>>>
>>> On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> ... but the error is not "because that already exists". See your stack
>>>> trace. It's because the definition is recursive. You define temp view
>>>> test1, create a second DF from it, and then redefine test1 as that result.
>>>> test1 depends on test1.
>>>>
>>>> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>
>>>>> Sean,
>>>>>
>>>>> The method name is very clear "createOrReplaceTempView" doesn't make
>>>>> any sense to throw an exception because this view already exists. Spark
>>>>> 3.2.x is breaking back compatibility with no reason or sense.
>>>>>
>>>>>
>>>>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> The error looks 'valid' - you define a temp view in terms of its own
>>>>>> previous version, which doesn't quite make sense - somewhere the new
>>>>>> definition depends on the old definition. I think it just correctly
>>>>>> surfaces as an error now,.
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>>>
>>>>>>> Hello team,
>>>>>>>
>>>>>>> I've found this issue while I was porting my project from Apache
>>>>>>> Spark 3.1.x to 3.2.x.
>>>>>>>
>>>>>>>
>>>>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>>>>
>>>>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>>>>
>>>>>>> Thank you so much
>>>>>>>
>>>>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>>>>> [info] at
>>>>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>>>>> [info] at
>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>>>>> [info] at
>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>>>>> [info] at
>>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>>>>> [info] at
>>>>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>>>>> [info] at
>>>>>>> scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>>>>> [info] at
>>>>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>>>>> [info] at
>>>>>>> scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> --
>>>>>>> Daniel Mantovani
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> --
>>>>> Daniel Mantovani
>>>>>
>>>>>
>>>
>>> --
>>>
>>> --
>>> Daniel Mantovani
>>>
>>>
>
> --
>
> --
> Daniel Mantovani
>
>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
I didn't post the SO issue, I've just found the same exception I'm facing
for Spark 3.2. Almaren Framework has a concept of create temporary views
with the name "__TABLE__".
Example, if you want to use SQL dialect to a DataFrame to join a
table/aggregation/apply a function whatever. Instead of you create a
temporary table you just use the "__TABLE__" alias. You don't really care
about the name of the table. You may use this "__TABLE__" approach in
different parts of your code.
Why can't I create or replace temporary views in different DataFrame with
the same name as before ?
On Mon, Dec 13, 2021 at 1:27 PM Sean Owen <sr...@gmail.com> wrote:
> If the issue is what you posted in SO, I think the stack trace explains it
> already. You want to avoid this recursive definition, which in general
> can't work.
> I think it's simply explicitly disallowed in all cases now, but, you
> should not be depending on this anyway - why can't this just be avoided?
>
> On Mon, Dec 13, 2021 at 10:06 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> Sean,
>>
>> https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2
>>
>> Just executing "sbt test" will reproduce the error, the same code works
>> for spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ?
>>
>> Thank you so much
>>
>>
>>
>> On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> ... but the error is not "because that already exists". See your stack
>>> trace. It's because the definition is recursive. You define temp view
>>> test1, create a second DF from it, and then redefine test1 as that result.
>>> test1 depends on test1.
>>>
>>> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>
>>>> Sean,
>>>>
>>>> The method name is very clear "createOrReplaceTempView" doesn't make
>>>> any sense to throw an exception because this view already exists. Spark
>>>> 3.2.x is breaking back compatibility with no reason or sense.
>>>>
>>>>
>>>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> The error looks 'valid' - you define a temp view in terms of its own
>>>>> previous version, which doesn't quite make sense - somewhere the new
>>>>> definition depends on the old definition. I think it just correctly
>>>>> surfaces as an error now,.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>>
>>>>>> Hello team,
>>>>>>
>>>>>> I've found this issue while I was porting my project from Apache
>>>>>> Spark 3.1.x to 3.2.x.
>>>>>>
>>>>>>
>>>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>>>
>>>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>>>
>>>>>> Thank you so much
>>>>>>
>>>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>>>> [info] at
>>>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>>>> [info] at
>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>>>> [info] at
>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>>>> [info] at
>>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>>>> [info] at
>>>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>>>> [info] at
>>>>>> scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>>>> [info] at
>>>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>>>> [info] at
>>>>>> scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>>>
>>>>>> --
>>>>>>
>>>>>> --
>>>>>> Daniel Mantovani
>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> --
>>>> Daniel Mantovani
>>>>
>>>>
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
If the issue is what you posted in SO, I think the stack trace explains it
already. You want to avoid this recursive definition, which in general
can't work.
I think it's simply explicitly disallowed in all cases now, but, you should
not be depending on this anyway - why can't this just be avoided?
On Mon, Dec 13, 2021 at 10:06 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
> Sean,
>
> https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2
>
> Just executing "sbt test" will reproduce the error, the same code works
> for spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ?
>
> Thank you so much
>
>
>
> On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sr...@gmail.com> wrote:
>
>> ... but the error is not "because that already exists". See your stack
>> trace. It's because the definition is recursive. You define temp view
>> test1, create a second DF from it, and then redefine test1 as that result.
>> test1 depends on test1.
>>
>> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
>> daniel.oliveira.mantovani@gmail.com> wrote:
>>
>>> Sean,
>>>
>>> The method name is very clear "createOrReplaceTempView" doesn't make
>>> any sense to throw an exception because this view already exists. Spark
>>> 3.2.x is breaking back compatibility with no reason or sense.
>>>
>>>
>>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> The error looks 'valid' - you define a temp view in terms of its own
>>>> previous version, which doesn't quite make sense - somewhere the new
>>>> definition depends on the old definition. I think it just correctly
>>>> surfaces as an error now,.
>>>>
>>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>>
>>>>> Hello team,
>>>>>
>>>>> I've found this issue while I was porting my project from Apache Spark
>>>>> 3.1.x to 3.2.x.
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>>
>>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>>
>>>>> Thank you so much
>>>>>
>>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>>> [info] at
>>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>>> [info] at
>>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>>> [info] at
>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>>> [info] at
>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>>> [info] at
>>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>>> [info] at
>>>>> scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>>> [info] at
>>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>>> [info] at
>>>>> scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>>
>>>>> --
>>>>>
>>>>> --
>>>>> Daniel Mantovani
>>>>>
>>>>>
>>>
>>> --
>>>
>>> --
>>> Daniel Mantovani
>>>
>>>
>
> --
>
> --
> Daniel Mantovani
>
>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
Sean,
https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2
Just executing "sbt test" will reproduce the error, the same code works for
spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ?
Thank you so much
On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sr...@gmail.com> wrote:
> ... but the error is not "because that already exists". See your stack
> trace. It's because the definition is recursive. You define temp view
> test1, create a second DF from it, and then redefine test1 as that result.
> test1 depends on test1.
>
> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> Sean,
>>
>> The method name is very clear "createOrReplaceTempView" doesn't make any
>> sense to throw an exception because this view already exists. Spark 3.2.x
>> is breaking back compatibility with no reason or sense.
>>
>>
>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> The error looks 'valid' - you define a temp view in terms of its own
>>> previous version, which doesn't quite make sense - somewhere the new
>>> definition depends on the old definition. I think it just correctly
>>> surfaces as an error now,.
>>>
>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>>> daniel.oliveira.mantovani@gmail.com> wrote:
>>>
>>>> Hello team,
>>>>
>>>> I've found this issue while I was porting my project from Apache Spark
>>>> 3.1.x to 3.2.x.
>>>>
>>>>
>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>>
>>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>>
>>>> Thank you so much
>>>>
>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>>> [info] at
>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>>> [info] at
>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>>> [info] at
>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>>> [info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>>> [info] at
>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>>> [info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>>
>>>> --
>>>>
>>>> --
>>>> Daniel Mantovani
>>>>
>>>>
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
... but the error is not "because that already exists". See your stack
trace. It's because the definition is recursive. You define temp view
test1, create a second DF from it, and then redefine test1 as that result.
test1 depends on test1.
On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
> Sean,
>
> The method name is very clear "createOrReplaceTempView" doesn't make any
> sense to throw an exception because this view already exists. Spark 3.2.x
> is breaking back compatibility with no reason or sense.
>
>
> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
>
>> The error looks 'valid' - you define a temp view in terms of its own
>> previous version, which doesn't quite make sense - somewhere the new
>> definition depends on the old definition. I think it just correctly
>> surfaces as an error now,.
>>
>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
>> daniel.oliveira.mantovani@gmail.com> wrote:
>>
>>> Hello team,
>>>
>>> I've found this issue while I was porting my project from Apache Spark
>>> 3.1.x to 3.2.x.
>>>
>>>
>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>>
>>> Do we have a bug for that in apache-spark or I need to create one ?
>>>
>>> Thank you so much
>>>
>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>>> [info] at
>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>>> [info] at
>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>>> [info] at
>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>>> [info] at
>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>> [info] at
>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>> [info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>> [info] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>> [info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>>
>>> --
>>>
>>> --
>>> Daniel Mantovani
>>>
>>>
>
> --
>
> --
> Daniel Mantovani
>
>
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Daniel de Oliveira Mantovani <da...@gmail.com>.
Sean,
The method name is very clear "createOrReplaceTempView" doesn't make any
sense to throw an exception because this view already exists. Spark 3.2.x
is breaking back compatibility with no reason or sense.
On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sr...@gmail.com> wrote:
> The error looks 'valid' - you define a temp view in terms of its own
> previous version, which doesn't quite make sense - somewhere the new
> definition depends on the old definition. I think it just correctly
> surfaces as an error now,.
>
> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantovani@gmail.com> wrote:
>
>> Hello team,
>>
>> I've found this issue while I was porting my project from Apache Spark
>> 3.1.x to 3.2.x.
>>
>>
>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>>
>> Do we have a bug for that in apache-spark or I need to create one ?
>>
>> Thank you so much
>>
>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
>> [info] org.apache.spark.sql.AnalysisException: Recursive view
>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
>> [info] at
>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
>> [info] at
>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
>> [info] at
>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
>> [info] at
>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
>> [info] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>> [info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
>> [info] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>> [info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>
>> --
>>
>> --
>> Daniel Mantovani
>>
>>
--
--
Daniel Mantovani
Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView
Posted by Sean Owen <sr...@gmail.com>.
The error looks 'valid' - you define a temp view in terms of its own
previous version, which doesn't quite make sense - somewhere the new
definition depends on the old definition. I think it just correctly
surfaces as an error now,.
On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantovani@gmail.com> wrote:
> Hello team,
>
> I've found this issue while I was porting my project from Apache Spark
> 3.1.x to 3.2.x.
>
>
> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi
>
> Do we have a bug for that in apache-spark or I need to create one ?
>
> Thank you so much
>
> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED ***
> [info] org.apache.spark.sql.AnalysisException: Recursive view
> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`)
> [info] at
> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045)
> [info] at
> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515)
> [info] at
> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522)
> [info] at
> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522)
> [info] at scala.collection.Iterator.foreach(Iterator.scala:941)
> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
> [info] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> [info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> [info] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> [info] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>
> --
>
> --
> Daniel Mantovani
>
>