You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Said (Jira)" <ji...@apache.org> on 2023/08/02 16:37:00 UTC
[jira] [Updated] (SPARK-44638) Unable to read from JDBC data sources when using custom schema containing varchar
[ https://issues.apache.org/jira/browse/SPARK-44638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Said updated SPARK-44638:
---------------------------------
Description:
When querying the data from JDBC databases with custom schema containing varchar I got this error :
{code:java}
[23/07/14 06:12:19 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1) (172.30.3.76 executor 1): java.sql.SQLException: Unsupported type varchar(100) at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedJdbcTypeError(QueryExecutionErrors.scala:818) 23/07/14 06:12:21 INFO TaskSetManager: Lost task 0.1 in stage 1.0 (TID 2) on 172.30.3.76, executor 0: java.sql.SQLException (Unsupported type varchar(100)){code}
Code example:
{code:java}
CUSTOM_SCHEMA="ID Integer, NAME VARCHAR(100)"
df = spark.read.format("jdbc")
.option("url", "jdbc:oracle:thin:@0.0.0.0:1521:db")
.option("driver", "oracle.jdbc.OracleDriver")
.option("dbtable", "table")
.option("customSchema", CUSTOM_SCHEMA)
.option("user", "user")
.option("password", "password")
.load()
df.show(){code}
I tried to set {{spark.sql.legacy.charVarcharAsString = true}} to restore the behavior before Spark 3.1 but it doesn't help.
The issue occurs in version 3.1.0 and above. I believe that this issue is caused by https://issues.apache.org/jira/browse/SPARK-33480
was:
When querying the data from JDBC databases with custom schema containing varchar I got this error :
{code:java}
[23/07/14 06:12:19 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1) (172.30.3.76 executor 1): java.sql.SQLException: Unsupported type varchar(100) at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedJdbcTypeError(QueryExecutionErrors.scala:818) 23/07/14 06:12:21 INFO TaskSetManager: Lost task 0.1 in stage 1.0 (TID 2) on 172.30.3.76, executor 0: java.sql.SQLException (Unsupported type varchar(100)){code}
Code example:
{code:java}
CUSTOM_SCHEMA="ID Integer, NAME VARCHAR(100)"
df = spark.read.format("jdbc")
.option("url", "jdbc:oracle:thin:@10.128.0.30:1521:db")
.option("driver", "oracle.jdbc.OracleDriver")
.option("dbtable", "table")
.option("customSchema", CUSTOM_SCHEMA)
.option("user", "user")
.option("password", "password")
.load()
df.show(){code}
I tried to set {{spark.sql.legacy.charVarcharAsString = true}} to restore the behavior before Spark 3.1 but it doesn't help.
The issue occurs in version 3.1.0 and above. I believe that this issue is caused by https://issues.apache.org/jira/browse/SPARK-33480
> Unable to read from JDBC data sources when using custom schema containing varchar
> ---------------------------------------------------------------------------------
>
> Key: SPARK-44638
> URL: https://issues.apache.org/jira/browse/SPARK-44638
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.0, 3.2.4, 3.3.2, 3.4.1
> Reporter: Michael Said
> Priority: Critical
>
> When querying the data from JDBC databases with custom schema containing varchar I got this error :
> {code:java}
> [23/07/14 06:12:19 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1) (172.30.3.76 executor 1): java.sql.SQLException: Unsupported type varchar(100) at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedJdbcTypeError(QueryExecutionErrors.scala:818) 23/07/14 06:12:21 INFO TaskSetManager: Lost task 0.1 in stage 1.0 (TID 2) on 172.30.3.76, executor 0: java.sql.SQLException (Unsupported type varchar(100)){code}
> Code example:
> {code:java}
> CUSTOM_SCHEMA="ID Integer, NAME VARCHAR(100)"
> df = spark.read.format("jdbc")
> .option("url", "jdbc:oracle:thin:@0.0.0.0:1521:db")
> .option("driver", "oracle.jdbc.OracleDriver")
> .option("dbtable", "table")
> .option("customSchema", CUSTOM_SCHEMA)
> .option("user", "user")
> .option("password", "password")
> .load()
> df.show(){code}
> I tried to set {{spark.sql.legacy.charVarcharAsString = true}} to restore the behavior before Spark 3.1 but it doesn't help.
> The issue occurs in version 3.1.0 and above. I believe that this issue is caused by https://issues.apache.org/jira/browse/SPARK-33480
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org