[jira] [Updated] (SPARK-24669) Managed table was not cleared of path after drop database cascade

["Dongjoon Hyun (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-24669:
----------------------------------
    Fix Version/s:     (was: 2.4.2)
                   2.4.1

> Managed table was not cleared of path after drop database cascade
> -----------------------------------------------------------------
>
>                 Key: SPARK-24669
>                 URL: https://issues.apache.org/jira/browse/SPARK-24669
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0, 2.3.1
>            Reporter: Dong Jiang
>            Assignee: Udbhav Agrawal
>            Priority: Major
>             Fix For: 2.3.4, 2.4.1, 3.0.0
>
>
> I can do the following in sequence
> # Create a managed table using path options
> # Drop the table via dropping the parent database cascade
> # Re-create the database and table with a different path
> # The new table shows data from the old path, not the new path
> {code}
> echo "first" > /tmp/first.csv
> echo "second" > /tmp/second.csv
> spark-shell
> spark.version
> res0: String = 2.3.0
> spark.sql("create database foo")
> spark.sql("create table foo.first (id string) using csv options (path='/tmp/first.csv')")
> spark.table("foo.first").show()
> +-----+
> |   id|
> +-----+
> |first|
> +-----+
> spark.sql("drop database foo cascade")
> spark.sql("create database foo")
> spark.sql("create table foo.first (id string) using csv options (path='/tmp/second.csv')")
> "note, the path is different now, pointing to second.csv, but still showing data from first file"
> spark.table("foo.first").show()
> +-----+
> |   id|
> +-----+
> |first|
> +-----+
> "now, if I drop the table explicitly, instead of via dropping database cascade, then it will be the correct result"
> spark.sql("drop table foo.first")
> spark.sql("create table foo.first (id string) using csv options (path='/tmp/second.csv')")
> spark.table("foo.first").show()
> +------+
> |    id|
> +------+
> |second|
> +------+
> {code}
> Same sequence failed in 2.3.1 as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org