You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kevinshin (Jira)" <ji...@apache.org> on 2023/02/24 05:53:00 UTC

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

     [ https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

kevinshin updated SPARK-42550:
------------------------------
    Description: 
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during execution, the table's directory will be deleted.  this is not happen in spark 3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 6000044164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}

  was:
{color:#4c9aff}when a  `*INSERT* OVERWRITE *TABLE`* statment faild during execution, the table's directory will be deleted.  this is not happen in spark 3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 *int*) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 6000044164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* test.spark32_overwrite2;    -- {color:#de350b}*this will got Casting overflow exception*{color}

{color:#4c9aff}and then :{color}

*select* *** *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}


> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---------------------------------------------------------------
>
>                 Key: SPARK-42550
>                 URL: https://issues.apache.org/jira/browse/SPARK-42550
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.2.3
>         Environment: spark 3.2.3 / HDP 3.1.4
>            Reporter: kevinshin
>            Priority: Critical
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during execution, the table's directory will be deleted.  this is not happen in spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 6000044164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}the table's directory is losted.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org