[jira] [Commented] (SPARK-53348) Always persist ANSI value when creating a view or assume it when querying if not stored

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-53348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016358#comment-18016358 ] 

Dongjoon Hyun commented on SPARK-53348:
---------------------------------------

This is reverted from `branch-4.0` first due to the non-ANSI CI failures. Previously, both `master` and `branch-4.0` non-ANSI CIs are broken.

- https://github.com/apache/spark/commit/f3c2d39f32c9122bf0e496082b34572185ce8f38
- https://github.com/apache/spark/actions/runs/17197538374/job/48928049245

{code}
[info] - Default ANSI value is stored for views *** FAILED *** (18 milliseconds)
[info]   "[fals]e" did not equal "[tru]e" (DefaultANSIValueSuite.scala:104)
[info]   Analysis:
[info]   "[fals]e" -> "[tru]e"
{code}

> Always persist ANSI value when creating a view or assume it when querying if not stored
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-53348
>                 URL: https://issues.apache.org/jira/browse/SPARK-53348
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Mihailo Aleksic
>            Assignee: Mihailo Aleksic
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> I propose that we always store ANSI value when creating a view because otherwise users can be affected by unwanted behavior. For example if user creates a view on version that has ANSI = false by default he expects this not to fail.
> CREATE VIEW view AS SELECT CAST('abc' AS INT) AS a;
> SELECT * FROM view;
> But if user queries the view on the version which has ANSI = true by default, above query is going to fail (because when we don't store the value, and we store it only if explicitly set, we use the default one). Number of this and similar use cases is huge, because ANSI impact area is huge and thus I propose that we always store the value.
> If the value is not stored, I propose that we use createVersion field to determine whether the ANSI value should be true (Spark 4.0.0 and above) or false (lower than Spark 4.0.0). If the createVersion field wasn't stored during view creation, I propose that we assume that the ANSI = false because number of those views is incomparable larger than the ones expecting ANSI = true



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org