[jira] [Updated] (SPARK-10756) DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence

["Amit (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit updated SPARK-10756:
-------------------------
    Description: 
DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence. 

Whenever it goes to persist dataframe it checks for the table existence by using query "SELECT 1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the exception thrown by teradata for keyword is understood as exception for table not exist in Spark and then Spark runs the create table command irrespective of table was present. 

So Create table command execution fails with the exception of table already exist hence saving of data frame fails 

Below is the method of JDBCUtils class

/**
   * Returns true if the table already exists in the JDBC database.
   */
  def tableExists(conn: Connection, table: String): Boolean = {
    // Somewhat hacky, but there isn't a good way to identify whether a table exists for all
    // SQL database systems, considering "table" could also include the database name.
    Try(conn.prepareStatement(s"SELECT 1 FROM $table LIMIT 1").executeQuery().next()).isSuccess
  }

In case of teradata, It returns false for every save/write operation irrespective of the fact that table was present.

  was:
DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence. 

Whenever it goes to persist dataframe it checks for the table existence by using query "SELECT 1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the exception thrown by teradata for keyword is understood as exception for table not exist in Spark and then Spark runs the create table command irrespective of table was present. 

So Create table command execution fails with the exception of table already exist hence saving of data frame fails 


> DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-10756
>                 URL: https://issues.apache.org/jira/browse/SPARK-10756
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Amit
>
> DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence. 
> Whenever it goes to persist dataframe it checks for the table existence by using query "SELECT 1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the exception thrown by teradata for keyword is understood as exception for table not exist in Spark and then Spark runs the create table command irrespective of table was present. 
> So Create table command execution fails with the exception of table already exist hence saving of data frame fails 
> Below is the method of JDBCUtils class
> /**
>    * Returns true if the table already exists in the JDBC database.
>    */
>   def tableExists(conn: Connection, table: String): Boolean = {
>     // Somewhat hacky, but there isn't a good way to identify whether a table exists for all
>     // SQL database systems, considering "table" could also include the database name.
>     Try(conn.prepareStatement(s"SELECT 1 FROM $table LIMIT 1").executeQuery().next()).isSuccess
>   }
> In case of teradata, It returns false for every save/write operation irrespective of the fact that table was present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org