You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marina Krasilnikova (Jira)" <ji...@apache.org> on 2023/01/11 19:09:00 UTC
[jira] [Created] (SPARK-41990) Filtering by composite field name like `field name` doesn't work with pushDownPredicate = true
Marina Krasilnikova created SPARK-41990:
-------------------------------------------
Summary: Filtering by composite field name like `field name` doesn't work with pushDownPredicate = true
Key: SPARK-41990
URL: https://issues.apache.org/jira/browse/SPARK-41990
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3.1, 3.3.0
Reporter: Marina Krasilnikova
Suppose we have some table in postgresql with field `Last Name` The following code results in error
Dataset<Row> dataset = sparkSession.read()
.format("jdbc")
.option("url", myUrl)
.option("dbtable", "myTable")
.option("user", "myUser")
.option("password", "muPassword")
.load();
dataset.where("`Last Name`='Tessel'").show(); //error
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
Syntax error at or near 'Name': extra input 'Name'(line 1, pos 5)
== SQL ==
Last Name
-----^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseMultipartIdentifier(ParseDriver.scala:67)
at org.apache.spark.sql.connector.expressions.LogicalExpressions$.parseReference(expressions.scala:40)
at org.apache.spark.sql.connector.expressions.FieldReference$.apply(expressions.scala:368)
at org.apache.spark.sql.sources.IsNotNull.toV2(filters.scala:262)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1(JDBCRelation.scala:278)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1$adapted(JDBCRelation.scala:278)
But if we set pushDownPredicate to false everything works fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org