You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org> on 2025/07/16 19:31:09 UTC

[PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

PetarVasiljevic-DB opened a new pull request, #51519:
URL: https://github.com/apache/spark/pull/51519

### What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/50921, Join pushdown was added for DSv2 and it was only enabled for H2 dialect.
With this PR, I am enabling DSv2 join pushdown for Oracle connector as well.

For this purpose, `OracleDialect` has now `supportsJoin` equal to true.
Also, changed SQL query generation to use `tableOrQuery` method instead of `options.tableOrQuery`.

The rest of the change is test only:
* Created new integration trait `JDBCJoinPushdownIntegrationSuite` that can be used for testing other connectors as well
* Added `OracleJoinPushdownIntegrationSuite` as the first implementation of the trait
* Added new test utils trait `V2PushdownTestUtils`. Extracted helper methods from `V2JDBCTest` into this trait. Now, both `V2JDBCTest` and `JDBCJoinPushdownIntegrationSuite` are inheriting `V2PushdownTestUtils`

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?
Yes, inner joins will be pushed down to Oracle data source.
Previously, Spark SQL query
```
SELECT tbl1.id, tbl1.name, tbl2.id
FROM oracleCatalog.tbl1 t1
JOIN oracleCatalog.tbl2 t2
ON t1.id = t2.id + 1
```
would produce the following Optimized plan:

```
== Optimized Logical Plan ==
Join Inner, (id#0 = (id#1 + 1))
:- Filter isnotnull(id#0)
: +- RelationV2[id#0] oracleCatalog.tbl1
+- Filter isnotnull(id#1, name#2)
+- RelationV2[id#1, name#2] oracleCatalog.tbl2
```

Now, with join pushdown enabled, the plan would be:

```
Project [ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c#3 AS id#0, ID#4 AS id#1, NAME#5 AS name#2]
+- RelationV2[ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c#3, ID#4, NAME#5] oracleCatalog.tbl1
```

When join is pushed down, the physical plan will contain `PushedJoins` information, which is the array of all the tables joined. For example, in the above case it would be:

```
PushedJoins: [oracleCatalog.tbl1, oracleCatalog.tbl2]
```

The generated SQL query would be:
```
SELECT
"ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c",
"ID",
"NAME"
FROM
(
SELECT
"ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c",
"ID",
"NAME"
FROM
(
SELECT
"ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c",
"ID",
"NAME"
FROM
(
SELECT
"ID" AS "ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c",
"NAME"
FROM
"SYSTEM"."TBL1"
WHERE
("ID" IS NOT NULL)
) join_subquery_4
INNER JOIN (
SELECT
"ID"
FROM
"SYSTEM"."TBL2"
WHERE
("ID" IS NOT NULL)
) join_subquery_5 ON "ID_974bb0c2_a32c_4d5b_b6ee_745efa1f3a0c" = "ID"
)
) SPARK_GEN_SUBQ_30
```

<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.

If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->

### How was this patch tested?
New tests.

### Was this patch authored or co-authored using generative AI tooling?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219830334


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   We are introducing a state for these util methods. Can we avoid that?
   If we can make util methods stateless, that would be better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213639068


##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcSQLQueryBuilder.scala:
##########
@@ -208,7 +208,7 @@ class JdbcSQLQueryBuilder(dialect: JdbcDialect, options: JDBCOptions) {
 
   // If join has been pushed down, reuse join query as a subquery. Otherwise, fallback to
   // what is provided in options.
-  private def tableOrQuery = joinQuery.getOrElse(options.tableOrQuery)
+  def tableOrQuery: String = joinQuery.getOrElse(options.tableOrQuery)

Review Comment:
   If you need to access it from derived suites, but does not want to make it overridable, you can use `protected final`
   ```suggestion
     def tableOrQuery: String = joinQuery.getOrElse(options.tableOrQuery)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213489337


##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala:
##########
@@ -233,7 +233,7 @@ private case class OracleDialect() extends JdbcDialect with SQLConfHelper with N
     extends JdbcSQLQueryBuilder(dialect, options) {
 
     override def build(): String = {
-      val selectStmt = s"SELECT $hintClause$columnList FROM ${options.tableOrQuery}" +
+      val selectStmt = s"SELECT $hintClause$columnList FROM $tableOrQuery" +

Review Comment:
   tableOrQuery will resolve to the table option then



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2221316782


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,603 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes, StructField, StructType}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespace: String = "join_schema"
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(SQLConf.ANSI_ENABLED.key, "true")
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+    .set(s"spark.sql.catalog.$catalogName.caseSensitive", "false")
+
+  private def catalogAndNamespace = s"$catalogName.${caseConvert(namespace)}"
+  private def casedJoinTableName1 = caseConvert(joinTableName1)
+  private def casedJoinTableName2 = caseConvert(joinTableName2)
+
+  def qualifyTableName(tableName: String): String = {
+    val fullyQualifiedCasedNamespace = jdbcDialect.quoteIdentifier(caseConvert(namespace))
+    val fullyQualifiedCasedTableName = jdbcDialect.quoteIdentifier(caseConvert(tableName))
+    s"$fullyQualifiedCasedNamespace.$fullyQualifiedCasedTableName"
+  }
+
+  def quoteSchemaName(schemaName: String): String =
+    jdbcDialect.quoteIdentifier(caseConvert(namespace))
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(identifier: String): String = identifier
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  protected val integerType = DataTypes.IntegerType
+
+  protected val stringType = DataTypes.StringType
+
+  protected val decimalType = DataTypes.createDecimalType(10, 2)
+
+  /**
+   * This method should cover the following:
+   * <ul>
+   *   <li>Create the schema where testing tables will be stored.
+   *   <li>Create the testing tables {@code joinTableName1} and {@code joinTableName2}
+   *   in above schema.
+   *   <li>Populate the tables with the data.
+   * </ul>
+   */
+  def dataPreparation(): Unit = {
+    schemaPreparation()
+    tablePreparation()
+    fillJoinTables()
+  }
+
+  def schemaPreparation(): Unit = {
+    withConnection {conn =>
+      conn
+        .prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${quoteSchemaName(namespace)}")
+        .executeUpdate()
+    }
+  }
+
+  def tablePreparation(): Unit = {
+    withConnection{ conn =>
+      conn.prepareStatement(
+        s"""CREATE TABLE $fullyQualifiedTableName1 (
+           |  ID ${getJDBCTypeString(integerType)},
+           |  AMOUNT ${getJDBCTypeString(decimalType)},
+           |  ADDRESS ${getJDBCTypeString(stringType)}
+           |)""".stripMargin
+      ).executeUpdate()
+
+      conn.prepareStatement(
+        s"""CREATE TABLE $fullyQualifiedTableName2 (
+           |  ID ${getJDBCTypeString(integerType)},
+           |  NEXT_ID ${getJDBCTypeString(integerType)},
+           |  SALARY ${getJDBCTypeString(decimalType)},
+           |  SURNAME ${getJDBCTypeString(stringType)}
+           |)""".stripMargin
+      ).executeUpdate()
+    }
+  }
+
+  private val random = new java.util.Random(42)
+
+  private val table1Data = (1 to 100).map { i =>
+    val id = i % 11
+    val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+      .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+    val address = s"address_$i"
+    (id, amount, address)
+  }
+
+  private val table2Data = (1 to 100).map { i =>
+    val id = (i % 17)
+    val next_id = (id + 1) % 17
+    val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+      .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+    val surname = s"surname_$i"
+    (id, next_id, salary, surname)
+  }
+
+  def fillJoinTables(): Unit = {
+    withConnection { conn =>
+      val insertStmt1 = conn.prepareStatement(
+        s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+      )
+      table1Data.foreach { case (id, amount, address) =>
+        insertStmt1.setInt(1, id)
+        insertStmt1.setBigDecimal(2, amount.bigDecimal)
+        insertStmt1.setString(3, address)
+        insertStmt1.addBatch()
+      }
+      insertStmt1.executeBatch()
+      insertStmt1.close()
+
+      val insertStmt2 = conn.prepareStatement(
+        s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+      )
+      table2Data.foreach { case (id, next_id, salary, surname) =>
+        insertStmt2.setInt(1, id)
+        insertStmt2.setInt(2, next_id)
+        insertStmt2.setBigDecimal(3, salary.bigDecimal)
+        insertStmt2.setString(4, surname)
+        insertStmt2.addBatch()
+      }
+      insertStmt2.executeBatch()
+      insertStmt2.close()
+
+    }
+  }
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   since we have `JdbcDialect` instance in this suite, we can get these `supportsXYZ` values from it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213639068


##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcSQLQueryBuilder.scala:
##########
@@ -208,7 +208,7 @@ class JdbcSQLQueryBuilder(dialect: JdbcDialect, options: JDBCOptions) {
 
   // If join has been pushed down, reuse join query as a subquery. Otherwise, fallback to
   // what is provided in options.
-  private def tableOrQuery = joinQuery.getOrElse(options.tableOrQuery)
+  def tableOrQuery: String = joinQuery.getOrElse(options.tableOrQuery)

Review Comment:
   If you need to access it from derived suites, but does not want to make it overridable, you can use `protected final`
   ```suggestion
     protected final def tableOrQuery: String = joinQuery.getOrElse(options.tableOrQuery)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213667785


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {

Review Comment:
   Can we make this suite more generic and to decouple it from JDBC to make it reusable by other non JDBC connectors?
   
   Maybe some class hieararchy like:
   - JoinPushdownIntegrationSuiteBase
   - JDBCJoinPushdownIntegrationSuiteBase extends JoinPushdownIntegrationSuiteBase
   - OracleJoinPushdownIntegrationSuiteBase extends JDBCJoinPushdownIntegrationSuiteBase
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215518842


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {

Review Comment:
   I am ok with that, that cause a little bit extra work for reviewers, but that is fine. Just have in mind we need another layer of abstraction, so we can easier do a followup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215644138


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"

Review Comment:
   Not really? If you have duplicate tables, like in my case, it's not asserted that it is shown 2 times in explain.
   With string, we are doing it.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2216272045


##########
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/join/OracleJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2.join
+
+import java.sql.Connection
+import java.util.Locale
+
+import org.apache.spark.sql.jdbc.{DockerJDBCIntegrationSuite, JdbcDialect, OracleDatabaseOnDocker, OracleDialect}
+import org.apache.spark.sql.jdbc.v2.JDBCV2JoinPushdownIntegrationSuiteBase
+import org.apache.spark.sql.types.DataTypes
+import org.apache.spark.tags.DockerTest
+
+/**
+ * The following are the steps to test this:
+ *
+ * 1. Choose to use a prebuilt image or build Oracle database in a container
+ *    - The documentation on how to build Oracle RDBMS in a container is at
+ *      https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
+ *    - Official Oracle container images can be found at https://container-registry.oracle.com
+ *    - Trustable and streamlined Oracle Database Free images can be found on Docker Hub at
+ *      https://hub.docker.com/r/gvenzl/oracle-free
+ *      see also https://github.com/gvenzl/oci-oracle-free
+ * 2. Run: export ORACLE_DOCKER_IMAGE_NAME=image_you_want_to_use_for_testing
+ *    - Example: export ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:latest
+ * 3. Run: export ENABLE_DOCKER_INTEGRATION_TESTS=1
+ * 4. Start docker: sudo service docker start
+ *    - Optionally, docker pull $ORACLE_DOCKER_IMAGE_NAME
+ * 5. Run Spark integration tests for Oracle with: ./build/sbt -Pdocker-integration-tests
+ *    "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite"
+ *
+ * A sequence of commands to build the Oracle Database Free container image:
+ *  $ git clone https://github.com/oracle/docker-images.git
+ *  $ cd docker-images/OracleDatabase/SingleInstance/dockerfiles0
+ *  $ ./buildContainerImage.sh -v 23.4.0 -f
+ *  $ export ORACLE_DOCKER_IMAGE_NAME=oracle/database:23.4.0-free
+ *
+ * This procedure has been validated with Oracle Database Free version 23.4.0,
+ * and with Oracle Express Edition versions 18.4.0 and 21.4.0
+ */
+@DockerTest
+class OracleJoinPushdownIntegrationSuite
+  extends DockerJDBCIntegrationSuite
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  override val namespace: String = "SYSTEM"
+
+  override val db = new OracleDatabaseOnDocker
+
+  override val url = db.getJdbcUrl(dockerIp, externalPort)
+
+  override val jdbcDialect: JdbcDialect = OracleDialect()
+
+  override val integerType = DataTypes.createDecimalType(10, 0)
+
+  override def caseConvert(tableName: String): String = tableName.toUpperCase(Locale.ROOT)
+
+  override def schemaPreparation(): Unit = {}
+
+  // This method comes from DockerJDBCIntegrationSuite
+  override def dataPreparation(connection: Connection): Unit = {
+    super.dataPreparation()

Review Comment:
   so this `override` is a noop?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213655771


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {

Review Comment:
   ```suggestion
       extends QueryTest
       with SharedSparkSession
       with V2JDBCPushdownTestUtils {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219806738


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   Is this one resolved?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219911746


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   Well it is not stateless since we are extending `ExplainSuiteHelper`, right? So I don't really think this is problematic. wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213670195


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   Can this make some flakiness?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213657745


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None

Review Comment:
   Can this be a val like other parameters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2216407577


##########
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/join/OracleJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2.join
+
+import java.sql.Connection
+import java.util.Locale
+
+import org.apache.spark.sql.jdbc.{DockerJDBCIntegrationSuite, JdbcDialect, OracleDatabaseOnDocker, OracleDialect}
+import org.apache.spark.sql.jdbc.v2.JDBCV2JoinPushdownIntegrationSuiteBase
+import org.apache.spark.sql.types.DataTypes
+import org.apache.spark.tags.DockerTest
+
+/**
+ * The following are the steps to test this:
+ *
+ * 1. Choose to use a prebuilt image or build Oracle database in a container
+ *    - The documentation on how to build Oracle RDBMS in a container is at
+ *      https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
+ *    - Official Oracle container images can be found at https://container-registry.oracle.com
+ *    - Trustable and streamlined Oracle Database Free images can be found on Docker Hub at
+ *      https://hub.docker.com/r/gvenzl/oracle-free
+ *      see also https://github.com/gvenzl/oci-oracle-free
+ * 2. Run: export ORACLE_DOCKER_IMAGE_NAME=image_you_want_to_use_for_testing
+ *    - Example: export ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:latest
+ * 3. Run: export ENABLE_DOCKER_INTEGRATION_TESTS=1
+ * 4. Start docker: sudo service docker start
+ *    - Optionally, docker pull $ORACLE_DOCKER_IMAGE_NAME
+ * 5. Run Spark integration tests for Oracle with: ./build/sbt -Pdocker-integration-tests
+ *    "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite"
+ *
+ * A sequence of commands to build the Oracle Database Free container image:
+ *  $ git clone https://github.com/oracle/docker-images.git
+ *  $ cd docker-images/OracleDatabase/SingleInstance/dockerfiles0
+ *  $ ./buildContainerImage.sh -v 23.4.0 -f
+ *  $ export ORACLE_DOCKER_IMAGE_NAME=oracle/database:23.4.0-free
+ *
+ * This procedure has been validated with Oracle Database Free version 23.4.0,
+ * and with Oracle Express Edition versions 18.4.0 and 21.4.0
+ */
+@DockerTest
+class OracleJoinPushdownIntegrationSuite
+  extends DockerJDBCIntegrationSuite
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  override val namespace: String = "SYSTEM"
+
+  override val db = new OracleDatabaseOnDocker
+
+  override val url = db.getJdbcUrl(dockerIp, externalPort)
+
+  override val jdbcDialect: JdbcDialect = OracleDialect()
+
+  override val integerType = DataTypes.createDecimalType(10, 0)
+
+  override def caseConvert(tableName: String): String = tableName.toUpperCase(Locale.ROOT)
+
+  override def schemaPreparation(): Unit = {}
+
+  // This method comes from DockerJDBCIntegrationSuite
+  override def dataPreparation(connection: Connection): Unit = {
+    super.dataPreparation()

Review Comment:
   no, it calls dataPreparation from `JDBCV2JoinPushdownIntegrationSuiteBase` that has default implementation.
   
   I just created dataPreparation() to be compatible for future non-jdbc connectors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213647871


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2JoinPushdownSuite.scala:
##########
@@ -17,21 +17,28 @@
 
 package org.apache.spark.sql.jdbc
 
-import java.sql.{Connection, DriverManager}
-import java.util.Properties
+import java.sql.Connection
 
 import org.apache.spark.SparkConf
-import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper, QueryTest}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, GlobalLimit, Join, Sort}
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
 import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
-import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.v2.{JDBCJoinPushdownIntegrationSuite, V2JDBCPushdownTestUtils}
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.util.Utils
 
-class JDBCV2JoinPushdownSuite extends QueryTest with SharedSparkSession with ExplainSuiteHelper {
+class JDBCV2JoinPushdownSuite

Review Comment:
   It is confusing to have `jdbc.JDBCV2JoinPushdownSuite` and `v2.JDBCJoinPushdownIntegrationSuite`. Can we name them somehow better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2214800976


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")
+
+  override val jdbcDialect: JdbcDialect = H2Dialect()
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set("spark.sql.catalog.h2", classOf[JDBCTableCatalog].getName)
+    .set("spark.sql.catalog.h2.url", url)
+    .set("spark.sql.catalog.h2.driver", "org.h2.Driver")
+    .set("spark.sql.catalog.h2.pushDownAggregate", "true")
+    .set("spark.sql.catalog.h2.pushDownLimit", "true")
+    .set("spark.sql.catalog.h2.pushDownOffset", "true")
+    .set("spark.sql.catalog.h2.pushDownJoin", "true")

Review Comment:
   shall we move most of the conf settings to the parent class? then here we only need
   ```
   override def sparkConf: SparkConf = super.sparkConf.set("spark.sql.catalog.h2.driver", "org.h2.Driver")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215536954


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {

Review Comment:
   Can you add docs here to describe a little bit what tables and schemas we need to create.



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {

Review Comment:
   To be easier to do followup later, can we remove connection argument and make each of these 3 make it's own connection, or if you want to save some calls, we can store connection somewhere



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(

Review Comment:
   We did not check whether column pruning actually happened, that can be done by checking leaf relation output



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aliases not supported in join pushdown") {
+    val sqlQuery = s"""
+      |SELECT a.id, bc.*
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN (
+      |  SELECT b.*, c.id AS c_id, c.amount AS c_amount
+      |  FROM $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |  JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |) bc ON bc.id = a.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with dataframe with duplicated columns") {
+    val df1 = sql(s"SELECT id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+    val df2 = sql(s"SELECT id, id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      df1.join(df2, "id").collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val joinDf = df1.join(df2, "id")
+
+      checkJoinPushed(
+        joinDf,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(joinDf, rows)
+    }
+  }
+
+  test("Test aggregate on top of 2-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkAggregateRemoved(df)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aggregate on top of multi-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id), min(c.id - 2)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test sort limit on top of join is pushed down") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), a.id, b.id
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |GROUP BY a.id, b.id
+      |ORDER BY a.id
+      |LIMIT 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(
+      SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkSortRemoved(df)

Review Comment:
   This one depends on scan builder implementor, we should make overridable member which tells whether sortLimit is supported and provide here that value:
   Like:
   ```suggestion
         checkSortRemoved(df, sortLimitPushdownSupported)
   ```



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"

Review Comment:
   You can provide here sequence of relations instead of doing string concat, it would be better fit for your signature.
   Also, can you name parameter here?
   ```suggestion
           expectedTables = Seq(
             s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
             s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
           )
   ```



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aliases not supported in join pushdown") {

Review Comment:
   Very good practice about testing negative scenarios



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215574986


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")

Review Comment:
   I don't have any preference here. I can make schema to not be an Option if it's a red flag here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2220507755


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   Is it better now? I moved these into `JDBCV2JoinPushdownIntegrationSuiteBase`. I still don't agree that ExplainSuite is stateless because if you go level deeper you will see states. It's not really pure util class.
   
   > It is beneficial to check whether filters are still in plan when pushdown is not supported
   
   However, I agree with this, so I changed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2221380089


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,603 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes, StructField, StructType}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespace: String = "join_schema"
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(SQLConf.ANSI_ENABLED.key, "true")
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+    .set(s"spark.sql.catalog.$catalogName.caseSensitive", "false")
+
+  private def catalogAndNamespace = s"$catalogName.${caseConvert(namespace)}"
+  private def casedJoinTableName1 = caseConvert(joinTableName1)
+  private def casedJoinTableName2 = caseConvert(joinTableName2)
+
+  def qualifyTableName(tableName: String): String = {
+    val fullyQualifiedCasedNamespace = jdbcDialect.quoteIdentifier(caseConvert(namespace))
+    val fullyQualifiedCasedTableName = jdbcDialect.quoteIdentifier(caseConvert(tableName))
+    s"$fullyQualifiedCasedNamespace.$fullyQualifiedCasedTableName"
+  }
+
+  def quoteSchemaName(schemaName: String): String =
+    jdbcDialect.quoteIdentifier(caseConvert(namespace))
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(identifier: String): String = identifier
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  protected val integerType = DataTypes.IntegerType
+
+  protected val stringType = DataTypes.StringType
+
+  protected val decimalType = DataTypes.createDecimalType(10, 2)
+
+  /**
+   * This method should cover the following:
+   * <ul>
+   *   <li>Create the schema where testing tables will be stored.
+   *   <li>Create the testing tables {@code joinTableName1} and {@code joinTableName2}
+   *   in above schema.
+   *   <li>Populate the tables with the data.
+   * </ul>
+   */
+  def dataPreparation(): Unit = {
+    schemaPreparation()
+    tablePreparation()
+    fillJoinTables()
+  }
+
+  def schemaPreparation(): Unit = {
+    withConnection {conn =>
+      conn
+        .prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${quoteSchemaName(namespace)}")
+        .executeUpdate()
+    }
+  }
+
+  def tablePreparation(): Unit = {
+    withConnection{ conn =>
+      conn.prepareStatement(
+        s"""CREATE TABLE $fullyQualifiedTableName1 (
+           |  ID ${getJDBCTypeString(integerType)},
+           |  AMOUNT ${getJDBCTypeString(decimalType)},
+           |  ADDRESS ${getJDBCTypeString(stringType)}
+           |)""".stripMargin
+      ).executeUpdate()
+
+      conn.prepareStatement(
+        s"""CREATE TABLE $fullyQualifiedTableName2 (
+           |  ID ${getJDBCTypeString(integerType)},
+           |  NEXT_ID ${getJDBCTypeString(integerType)},
+           |  SALARY ${getJDBCTypeString(decimalType)},
+           |  SURNAME ${getJDBCTypeString(stringType)}
+           |)""".stripMargin
+      ).executeUpdate()
+    }
+  }
+
+  private val random = new java.util.Random(42)
+
+  private val table1Data = (1 to 100).map { i =>
+    val id = i % 11
+    val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+      .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+    val address = s"address_$i"
+    (id, amount, address)
+  }
+
+  private val table2Data = (1 to 100).map { i =>
+    val id = (i % 17)
+    val next_id = (id + 1) % 17
+    val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+      .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+    val surname = s"surname_$i"
+    (id, next_id, salary, surname)
+  }
+
+  def fillJoinTables(): Unit = {
+    withConnection { conn =>
+      val insertStmt1 = conn.prepareStatement(
+        s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+      )
+      table1Data.foreach { case (id, amount, address) =>
+        insertStmt1.setInt(1, id)
+        insertStmt1.setBigDecimal(2, amount.bigDecimal)
+        insertStmt1.setString(3, address)
+        insertStmt1.addBatch()
+      }
+      insertStmt1.executeBatch()
+      insertStmt1.close()
+
+      val insertStmt2 = conn.prepareStatement(
+        s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+      )
+      table2Data.foreach { case (id, next_id, salary, surname) =>
+        insertStmt2.setInt(1, id)
+        insertStmt2.setInt(2, next_id)
+        insertStmt2.setBigDecimal(3, salary.bigDecimal)
+        insertStmt2.setString(4, surname)
+        insertStmt2.addBatch()
+      }
+      insertStmt2.executeBatch()
+      insertStmt2.close()
+
+    }
+  }
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   we have `supportsLimit`, `supportsOffset`, and `supportsJoin`. Others don't exist. I would leave it as it is as it can confuse someone why other variables don't use dialect.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215594311


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   That's a good point. I have materialized them. Also, added batch execution for JDBC statements, previously we were doing 100 network calls, now only 1.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219812558


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(

Review Comment:
   I see, ahahha, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219811828


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"

Review Comment:
   I meant to have same validation, but to make more friendly API, which accepts sequence of relations. We can still check whether all of those are in explain output. And, also, we can verify `ScanBuilderHolder` members easier than with arbitrary input string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on PR #51519:
URL: https://github.com/apache/spark/pull/51519#issuecomment-3101567551

   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215313460


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")
+
+  override val jdbcDialect: JdbcDialect = H2Dialect()
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set("spark.sql.catalog.h2", classOf[JDBCTableCatalog].getName)
+    .set("spark.sql.catalog.h2.url", url)
+    .set("spark.sql.catalog.h2.driver", "org.h2.Driver")
+    .set("spark.sql.catalog.h2.pushDownAggregate", "true")
+    .set("spark.sql.catalog.h2.pushDownLimit", "true")
+    .set("spark.sql.catalog.h2.pushDownOffset", "true")
+    .set("spark.sql.catalog.h2.pushDownJoin", "true")

Review Comment:
   Moved the common confs to base trait



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215768933


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aliases not supported in join pushdown") {
+    val sqlQuery = s"""
+      |SELECT a.id, bc.*
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN (
+      |  SELECT b.*, c.id AS c_id, c.amount AS c_amount
+      |  FROM $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |  JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |) bc ON bc.id = a.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with dataframe with duplicated columns") {
+    val df1 = sql(s"SELECT id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+    val df2 = sql(s"SELECT id, id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      df1.join(df2, "id").collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val joinDf = df1.join(df2, "id")
+
+      checkJoinPushed(
+        joinDf,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(joinDf, rows)
+    }
+  }
+
+  test("Test aggregate on top of 2-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkAggregateRemoved(df)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aggregate on top of multi-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id), min(c.id - 2)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test sort limit on top of join is pushed down") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), a.id, b.id
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |GROUP BY a.id, b.id
+      |ORDER BY a.id
+      |LIMIT 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(
+      SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkSortRemoved(df)

Review Comment:
   added `supportsXYZpushdown` in DataSourcePushdownTestUtils. Suites that extend it can override these values if some of the pushdowns are not supported



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213656521


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite

Review Comment:
   Do we need `Base` in name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213642901


##########
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala:
##########
@@ -1028,27 +945,6 @@ private[v2] trait V2JDBCTest extends SharedSparkSession with DockerIntegrationFu
     }
   }
 
-  private def checkAggregateRemoved(df: DataFrame): Unit = {
-    val aggregates = df.queryExecution.optimizedPlan.collect {
-      case agg: Aggregate => agg
-    }
-    assert(aggregates.isEmpty)
-  }
-
-  private def checkAggregatePushed(df: DataFrame, funcName: String): Unit = {

Review Comment:
   Good change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213507967


##########
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/join/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2.join
+
+import java.sql.Connection
+
+import org.apache.spark.sql.{DataFrame, QueryTest}
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.{DockerIntegrationFunSuite, JdbcDialect}
+import org.apache.spark.sql.jdbc.v2.V2JDBCPushdownTestUtils
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+import org.apache.spark.tags.DockerTest
+
+@DockerTest
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with DockerIntegrationFunSuite
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  val namespaceOpt: Option[String] = None
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  // Concrete suite must provide the dialect for its DB
+  def jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def fullyQualifiedTableName1: String = namespaceOpt
+    .map(namespace => s"$namespace.$joinTableName1").getOrElse(joinTableName1)
+
+  def fullyQualifiedTableName2: String = namespaceOpt
+    .map(namespace => s"$namespace.$joinTableName2").getOrElse(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  def dataPreparation(connection: Connection): Unit = {
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  id ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  amount ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  name ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  id ${getJDBCTypeString(DataTypes.LongType)},
+         |  salary ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  surname ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val name = s"name_$i"
+      (id, amount, name)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17).toLong
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, salary, surname)
+    }
+
+    // Use parameterized queries to handle different data types properly
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, name) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, name) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, name)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, salary, surname) VALUES (?, ?, ?)"
+    )
+    table2Data.foreach { case (id, salary, surname) =>
+      insertStmt2.setLong(1, id)
+      insertStmt2.setBigDecimal(2, salary.bigDecimal)
+      insertStmt2.setString(3, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  /**
+   * Runs the plan and makes sure the plans contains all of the keywords.
+   */
+  protected def checkKeywordsExistsInExplain(df: DataFrame, keywords: String*): Unit = {
+    val output = new java.io.ByteArrayOutputStream()
+    Console.withOut(output) {
+      df.explain(extended = true)
+    }
+    val normalizedOutput = output.toString.replaceAll("#\\d+", "#x")
+    for (key <- keywords) {
+      assert(normalizedOutput.contains(key), s"Expected keyword '$key' not found in explain output")
+    }
+  }
+
+  private def checkPushedInfo(df: DataFrame, expectedPlanFragment: String*): Unit = {
+    withSQLConf(SQLConf.MAX_METADATA_STRING_LENGTH.key -> "1000") {
+      df.queryExecution.optimizedPlan.collect {
+        case _: DataSourceV2ScanRelation =>
+          checkKeywordsExistsInExplain(df, expectedPlanFragment: _*)
+      }
+    }
+  }
+
+  private def checkJoinNotPushed(df: DataFrame): Unit = {
+    val joinNodes = df.queryExecution.optimizedPlan.collect {
+      case j: Join => j
+    }
+    assert(joinNodes.nonEmpty, "Join should not be pushed down")
+  }
+
+  private def checkJoinPushed(df: DataFrame, expectedTables: String*): Unit = {
+    val joinNodes = df.queryExecution.optimizedPlan.collect {
+      case j: Join => j
+    }
+    assert(joinNodes.isEmpty, "Join should be pushed down")
+    if (expectedTables.nonEmpty) {
+      checkPushedInfo(df, s"PushedJoins: [${expectedTables.mkString(", ")}]")
+    }
+  }
+
+  test("Test basic inner join pushdown with column pruning") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount, t2.salary
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    // Verify we have non-empty results
+    assert(rows.nonEmpty, "Join should produce non-empty results")
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+
+  test("Test join with additional filters") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount, t2.salary
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |WHERE t1.amount > 5000 AND t2.salary > 25000
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkFilterPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.name as name2
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join without condition should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |CROSS JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with complex condition") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount + t2.salary as total
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2
+      |ON t1.id = t2.id AND t1.amount > 1000
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with limit and order") {
+    // ORDER BY is used to have same ordering on Spark and database. Otherwise, different results
+    // could be returned.
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |ORDER BY t1.id, t1.name, t2.surname
+      |LIMIT 5
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkSortRemoved(df)
+      checkLimitRemoved(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with order by") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |ORDER BY t1.id, t1.name, t2.surname
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      // Order without limit is not supported in DSv2
+      checkSortRemoved(df, false)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with aggregation") {
+    val sqlQuery = s"""
+      |SELECT t1.id, COUNT(*), AVG(t1.amount), MAX(t2.salary)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |GROUP BY t1.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAggregateRemoved(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test left outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |LEFT JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test right outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |RIGHT JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test full outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |FULL OUTER JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with subquery should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, sub.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN (
+      |  SELECT id, surname FROM $catalogAndNamespace.${caseConvert(joinTableName2)}
+      |  WHERE salary > 25000
+      |) sub ON t1.id = sub.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with non-equality condition should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id > t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+}

Review Comment:
   anti join is not supported for pushdown, but yes, Spark has anti join



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2220523443


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   > I still don't agree that ExplainSuite is stateless because if you go level deeper you will see states
   Good catch, I did not see those ones. Thanks



##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   > I still don't agree that ExplainSuite is stateless because if you go level deeper you will see states
   
   Good catch, I did not see those ones. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219820797


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aliases not supported in join pushdown") {
+    val sqlQuery = s"""
+      |SELECT a.id, bc.*
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN (
+      |  SELECT b.*, c.id AS c_id, c.amount AS c_amount
+      |  FROM $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |  JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |) bc ON bc.id = a.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with dataframe with duplicated columns") {
+    val df1 = sql(s"SELECT id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+    val df2 = sql(s"SELECT id, id FROM $catalogAndNamespace.${caseConvert(joinTableName1)}")
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      df1.join(df2, "id").collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val joinDf = df1.join(df2, "id")
+
+      checkJoinPushed(
+        joinDf,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(joinDf, rows)
+    }
+  }
+
+  test("Test aggregate on top of 2-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkAggregateRemoved(df)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test aggregate on top of multi-way self join") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), min(a.id), min(c.id - 2)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test sort limit on top of join is pushed down") {
+    val sqlQuery = s"""
+      |SELECT min(a.id + b.id), a.id, b.id
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |GROUP BY a.id, b.id
+      |ORDER BY a.id
+      |LIMIT 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(
+      SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkSortRemoved(df)

Review Comment:
   I think it is more readable to provide it through parameter, since someone who reads code knows what's happening even without taking a look on `DataSourcePushdownTestUtils`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215508672


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")

Review Comment:
   It is maybe personal preference, but I like to explicitly put schema actually. It is a little bit confusing to me we have tests where we use just `catalog.table`, because it is unintuitive we use 2 part identifier for `catalog.table` instead of `schema.table`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215520855


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCPushdownTestUtils.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, GlobalLimit, Join, LocalLimit, Offset, Sample, Sort}
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+
+trait V2JDBCPushdownTestUtils extends ExplainSuiteHelper {

Review Comment:
   Nice, perfect



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213670789


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   Is 42 seed of Random function?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213668136


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {

Review Comment:
   it's already good indentation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213624758


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCPushdownTestUtils.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, GlobalLimit, Join, LocalLimit, Offset, Sample, Sort}
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+
+trait V2JDBCPushdownTestUtils extends ExplainSuiteHelper {

Review Comment:
   Out of v2 folder, in connector folder



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213624301


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCPushdownTestUtils.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, GlobalLimit, Join, LocalLimit, Offset, Sample, Sort}
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+
+trait V2JDBCPushdownTestUtils extends ExplainSuiteHelper {

Review Comment:
   What do you think about `DataSourcePushdownTestUtils`?



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCPushdownTestUtils.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, GlobalLimit, Join, LocalLimit, Offset, Sample, Sort}
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+
+trait V2JDBCPushdownTestUtils extends ExplainSuiteHelper {

Review Comment:
   Out of v2 folder



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2214801434


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"

Review Comment:
   seems the catalog name doesn't matter for the test cases, shall we just hardcode `jdbc_test` in the parent class?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215288775


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")

Review Comment:
   I've followed similar design as `V2JDBCTest` which has `namespaceOpt` as Option. For example, `MsSqlServerIntegrationSuite` and `MySQLIntegrationSuite` don't override it.
   
   I guess these tests will still work if we add a schema, I just wanted to be consistent with `V2JDBCTest`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215514821


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   Makes sense, however, I think it is better to materialize results as static rows so it would be more readable. Without debugger, developers won't know what are our test data, and it is hard to modify it as well later if we need it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215733952


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215733523


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }
+
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, address) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, address) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, address)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, next_id, salary, surname) VALUES (?, ?, ?, ?)"
+    )
+    table2Data.foreach { case (id, next_id, salary, surname) =>
+      insertStmt2.setInt(1, id)
+      insertStmt2.setInt(2, next_id)
+      insertStmt2.setBigDecimal(3, salary.bigDecimal)
+      insertStmt2.setString(4, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that 2-way join without condition should not have join pushed down") {
+    val sqlQuery =
+      s"""
+         |SELECT * FROM
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+         |$catalogAndNamespace.${caseConvert(joinTableName1)} b
+         |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  // Condition-less joins are not supported in join pushdown
+  test("Test that multi-way join without condition should not have join pushed down") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} b,
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} c
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with condition") {
+    val sqlQuery = s"""
+      |SELECT * FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with conditions") {
+    val sqlQuery = s"""
+      |SELECT * FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    assert(!rows.isEmpty)
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id + 2, b.id, b.amount FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b
+      |ON a.id = b.id + 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}, " +
+          s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test 2-way join with column pruning - different tables") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.next_id FROM
+      |$catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} b
+      |ON a.id = b.next_id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}," +
+          s" $catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkPushedInfo(df,
+        "PushedFilters: [ID IS NOT NULL, NEXT_ID IS NOT NULL, ID = NEXT_ID]")
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test multi-way self join with column pruning") {
+    val sqlQuery = s"""
+      |SELECT a.id, b.*, c.id, c.amount + a.amount
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} a
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} b ON b.id = a.id + 1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} c ON c.id = b.id - 1
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+
+      checkJoinPushed(

Review Comment:
   done now. UUIDs are used in column names so it's kind of hard to add expected schema.



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private lazy val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private lazy val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {

Review Comment:
   done, is it better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2220481733


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true

Review Comment:
   Hmm, it seems we only have util methods here?
   https://github.com/urosstan-db/spark/blob/1570206f58bc0858e8936b42df8f7c9b34b661c2/sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala#L33
   
   I would like to make it simpler, methods should check whether filter is removed or not, while in derived suites we can call them using:
   `checkFilterPushed(df, pushed = pushdownSupported && pushdownEnabled)`
   It is beneficial to check whether filters are still in plan when pushdown is not supported, sometimes pushdown rules/strategies just swallow nodes from plan making correctness issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan closed pull request #51519: [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector
URL: https://github.com/apache/spark/pull/51519


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213661100


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)

Review Comment:
   If someone overrides `qualifyTableName`, our val would be calculated in initialization of object, so would it use overriden method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213687732


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCPushdownTestUtils.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, GlobalLimit, Join, LocalLimit, Offset, Sample, Sort}
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+
+trait V2JDBCPushdownTestUtils extends ExplainSuiteHelper {

Review Comment:
   I've put it in `sql/connector`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2216475656


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,578 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes, StructField, StructType}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespace: String = "join_schema"
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace = s"$catalogName.$namespace"
+
+  def qualifyTableName(tableName: String): String = s"$namespace.$tableName"

Review Comment:
   yes, did the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219827282


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true
+
+
+  protected def checkSamplePushed(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsSamplePushdown) {
+      val sample = df.queryExecution.optimizedPlan.collect {
+        case s: Sample => s
+      }
+      if (pushed) {
+        assert(sample.isEmpty)
+      } else {
+        assert(sample.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkFilterPushed(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsFilterPushdown) {
+      val filter = df.queryExecution.optimizedPlan.collect {
+        case f: Filter => f
+      }
+      if (pushed) {
+        assert(filter.isEmpty)
+      } else {
+        assert(filter.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkLimitRemoved(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsLimitPushdown) {
+      val limit = df.queryExecution.optimizedPlan.collect {
+        case l: LocalLimit => l
+        case g: GlobalLimit => g
+      }
+      if (pushed) {
+        assert(limit.isEmpty)
+      } else {
+        assert(limit.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkLimitPushed(df: DataFrame, limit: Option[Int]): Unit = {
+    if (supportsLimitPushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            assert(v1.pushedDownOperators.limit == limit)
+        }
+      }
+    }
+  }
+
+  protected def checkColumnPruned(df: DataFrame, col: String): Unit = {
+    if (supportsColumnPruning) {
+      val scan = df.queryExecution.optimizedPlan.collectFirst {
+        case s: DataSourceV2ScanRelation => s
+      }.get
+      assert(scan.schema.names.sameElements(Seq(col)))
+    }
+  }
+
+  protected def checkAggregateRemoved(df: DataFrame): Unit = {
+    if (supportsAggregatePushdown) {
+      val aggregates = df.queryExecution.optimizedPlan.collect {
+        case agg: Aggregate => agg
+      }
+      assert(aggregates.isEmpty)
+    }
+  }
+
+  protected def checkAggregatePushed(df: DataFrame, funcName: String): Unit = {
+    if (supportsAggregatePushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case DataSourceV2ScanRelation(_, scan, _, _, _) =>
+          assert(scan.isInstanceOf[V1ScanWrapper])
+          val wrapper = scan.asInstanceOf[V1ScanWrapper]
+          assert(wrapper.pushedDownOperators.aggregation.isDefined)
+          val aggregationExpressions =
+            wrapper.pushedDownOperators.aggregation.get.aggregateExpressions()
+          assert(aggregationExpressions.exists { expr =>
+            expr.isInstanceOf[GeneralAggregateFunc] &&
+              expr.asInstanceOf[GeneralAggregateFunc].name() == funcName
+          })
+      }
+    }
+  }
+
+  protected def checkSortRemoved(
+      df: DataFrame,
+      pushed: Boolean = true): Unit = {
+    if (supportsSortPushdown) {
+      val sorts = df.queryExecution.optimizedPlan.collect {
+        case s: Sort => s
+      }
+
+      if (pushed) {
+        assert(sorts.isEmpty)
+      } else {
+        assert(sorts.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkOffsetRemoved(
+      df: DataFrame,
+      pushed: Boolean = true): Unit = {
+    if (supportsOffsetPushdown) {
+      val offsets = df.queryExecution.optimizedPlan.collect {
+        case o: Offset => o
+      }
+
+      if (pushed) {
+        assert(offsets.isEmpty)
+      } else {
+        assert(offsets.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkOffsetPushed(df: DataFrame, offset: Option[Int]): Unit = {
+    if (supportsOffsetPushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            assert(v1.pushedDownOperators.offset == offset)
+        }
+      }
+    }
+  }
+
+  protected def checkJoinNotPushed(df: DataFrame): Unit = {
+    if (supportsJoinPushdown) {
+      val joinNodes = df.queryExecution.optimizedPlan.collect {
+        case j: Join => j
+      }
+      assert(joinNodes.nonEmpty, "Join should not be pushed down")
+    }
+  }
+
+  protected def checkJoinPushed(df: DataFrame, expectedTables: String*): Unit = {
+    if (supportsJoinPushdown) {
+      val joinNodes = df.queryExecution.optimizedPlan.collect {
+        case j: Join => j
+      }
+      assert(joinNodes.isEmpty, "Join should be pushed down")
+      if (expectedTables.nonEmpty) {
+        checkPushedInfo(df, s"PushedJoins: [${expectedTables.mkString(", ")}]")
+      }
+    }
+  }
+
+  protected def checkPushedInfo(df: DataFrame, expectedPlanFragment: String*): Unit = {
+    withSQLConf(SQLConf.MAX_METADATA_STRING_LENGTH.key -> "1000") {
+      df.queryExecution.optimizedPlan.collect {
+        case _: DataSourceV2ScanRelation =>
+          checkKeywordsExistsInExplain(df, expectedPlanFragment: _*)
+      }
+    }
+  }
+
+  /**
+   * Check if the output schema of dataframe {@code df} is same as {@code schema}. There is one
+   * limitation: if expected schema name is empty, assertion on same names will be skipped.
+   * <br>
+   * For example, it is not really possible to use {@code checkPrunedColumns} for join pushdown,
+   * because in case of duplicate names, columns will have random UUID suffixes. For this reason,
+   * the best we can do is test that the size is same, and other fields beside names do match.
+   */
+  protected def checkPrunedColumnsDataTypeAndNullability(
+      df: DataFrame,
+      schema: StructType): Unit = {
+    if (supportsColumnPruning) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            val dfSchema = v1.readSchema()
+
+            assert(dfSchema.length == schema.length)
+            dfSchema.fields.zip(schema.fields).foreach { case (f1, f2) =>
+              if (f2.name.nonEmpty) {
+                assert(f1.name == f2.name)
+              }

Review Comment:
   Is there some better way to do this? It is very uncommon implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2219905863


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourcePushdownTestUtils.scala:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.{DataFrame, ExplainSuiteHelper}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.connector.expressions.aggregate.GeneralAggregateFunc
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, V1ScanWrapper}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+trait DataSourcePushdownTestUtils extends ExplainSuiteHelper {
+  protected val supportsSamplePushdown: Boolean = true
+
+  protected val supportsFilterPushdown: Boolean = true
+
+  protected val supportsLimitPushdown: Boolean = true
+
+  protected val supportsAggregatePushdown: Boolean = true
+
+  protected val supportsSortPushdown: Boolean = true
+
+  protected val supportsOffsetPushdown: Boolean = true
+
+  protected val supportsColumnPruning: Boolean = true
+
+  protected val supportsJoinPushdown: Boolean = true
+
+
+  protected def checkSamplePushed(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsSamplePushdown) {
+      val sample = df.queryExecution.optimizedPlan.collect {
+        case s: Sample => s
+      }
+      if (pushed) {
+        assert(sample.isEmpty)
+      } else {
+        assert(sample.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkFilterPushed(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsFilterPushdown) {
+      val filter = df.queryExecution.optimizedPlan.collect {
+        case f: Filter => f
+      }
+      if (pushed) {
+        assert(filter.isEmpty)
+      } else {
+        assert(filter.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkLimitRemoved(df: DataFrame, pushed: Boolean = true): Unit = {
+    if (supportsLimitPushdown) {
+      val limit = df.queryExecution.optimizedPlan.collect {
+        case l: LocalLimit => l
+        case g: GlobalLimit => g
+      }
+      if (pushed) {
+        assert(limit.isEmpty)
+      } else {
+        assert(limit.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkLimitPushed(df: DataFrame, limit: Option[Int]): Unit = {
+    if (supportsLimitPushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            assert(v1.pushedDownOperators.limit == limit)
+        }
+      }
+    }
+  }
+
+  protected def checkColumnPruned(df: DataFrame, col: String): Unit = {
+    if (supportsColumnPruning) {
+      val scan = df.queryExecution.optimizedPlan.collectFirst {
+        case s: DataSourceV2ScanRelation => s
+      }.get
+      assert(scan.schema.names.sameElements(Seq(col)))
+    }
+  }
+
+  protected def checkAggregateRemoved(df: DataFrame): Unit = {
+    if (supportsAggregatePushdown) {
+      val aggregates = df.queryExecution.optimizedPlan.collect {
+        case agg: Aggregate => agg
+      }
+      assert(aggregates.isEmpty)
+    }
+  }
+
+  protected def checkAggregatePushed(df: DataFrame, funcName: String): Unit = {
+    if (supportsAggregatePushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case DataSourceV2ScanRelation(_, scan, _, _, _) =>
+          assert(scan.isInstanceOf[V1ScanWrapper])
+          val wrapper = scan.asInstanceOf[V1ScanWrapper]
+          assert(wrapper.pushedDownOperators.aggregation.isDefined)
+          val aggregationExpressions =
+            wrapper.pushedDownOperators.aggregation.get.aggregateExpressions()
+          assert(aggregationExpressions.exists { expr =>
+            expr.isInstanceOf[GeneralAggregateFunc] &&
+              expr.asInstanceOf[GeneralAggregateFunc].name() == funcName
+          })
+      }
+    }
+  }
+
+  protected def checkSortRemoved(
+      df: DataFrame,
+      pushed: Boolean = true): Unit = {
+    if (supportsSortPushdown) {
+      val sorts = df.queryExecution.optimizedPlan.collect {
+        case s: Sort => s
+      }
+
+      if (pushed) {
+        assert(sorts.isEmpty)
+      } else {
+        assert(sorts.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkOffsetRemoved(
+      df: DataFrame,
+      pushed: Boolean = true): Unit = {
+    if (supportsOffsetPushdown) {
+      val offsets = df.queryExecution.optimizedPlan.collect {
+        case o: Offset => o
+      }
+
+      if (pushed) {
+        assert(offsets.isEmpty)
+      } else {
+        assert(offsets.nonEmpty)
+      }
+    }
+  }
+
+  protected def checkOffsetPushed(df: DataFrame, offset: Option[Int]): Unit = {
+    if (supportsOffsetPushdown) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            assert(v1.pushedDownOperators.offset == offset)
+        }
+      }
+    }
+  }
+
+  protected def checkJoinNotPushed(df: DataFrame): Unit = {
+    if (supportsJoinPushdown) {
+      val joinNodes = df.queryExecution.optimizedPlan.collect {
+        case j: Join => j
+      }
+      assert(joinNodes.nonEmpty, "Join should not be pushed down")
+    }
+  }
+
+  protected def checkJoinPushed(df: DataFrame, expectedTables: String*): Unit = {
+    if (supportsJoinPushdown) {
+      val joinNodes = df.queryExecution.optimizedPlan.collect {
+        case j: Join => j
+      }
+      assert(joinNodes.isEmpty, "Join should be pushed down")
+      if (expectedTables.nonEmpty) {
+        checkPushedInfo(df, s"PushedJoins: [${expectedTables.mkString(", ")}]")
+      }
+    }
+  }
+
+  protected def checkPushedInfo(df: DataFrame, expectedPlanFragment: String*): Unit = {
+    withSQLConf(SQLConf.MAX_METADATA_STRING_LENGTH.key -> "1000") {
+      df.queryExecution.optimizedPlan.collect {
+        case _: DataSourceV2ScanRelation =>
+          checkKeywordsExistsInExplain(df, expectedPlanFragment: _*)
+      }
+    }
+  }
+
+  /**
+   * Check if the output schema of dataframe {@code df} is same as {@code schema}. There is one
+   * limitation: if expected schema name is empty, assertion on same names will be skipped.
+   * <br>
+   * For example, it is not really possible to use {@code checkPrunedColumns} for join pushdown,
+   * because in case of duplicate names, columns will have random UUID suffixes. For this reason,
+   * the best we can do is test that the size is same, and other fields beside names do match.
+   */
+  protected def checkPrunedColumnsDataTypeAndNullability(
+      df: DataFrame,
+      schema: StructType): Unit = {
+    if (supportsColumnPruning) {
+      df.queryExecution.optimizedPlan.collect {
+        case relation: DataSourceV2ScanRelation => relation.scan match {
+          case v1: V1ScanWrapper =>
+            val dfSchema = v1.readSchema()
+
+            assert(dfSchema.length == schema.length)
+            dfSchema.fields.zip(schema.fields).foreach { case (f1, f2) =>
+              if (f2.name.nonEmpty) {
+                assert(f1.name == f2.name)
+              }

Review Comment:
   ideally I would do `dfSchema.equals(schema)` but I can't do it since dfSchema will have random names.
   
   I don't see a nicer way for doing this. We can remove the `name` check though so it is cleaner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "urosstan-db (via GitHub)" <gi...@apache.org>.

urosstan-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213667785


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {

Review Comment:
   Can we make this suite more generic and to decouple it from JDBC to make it reusable by other non JDBC connectors?
   
   Maybe some class hieararchy like:
   - `JoinPushdownIntegrationSuiteBase`
   - `JDBCJoinPushdownIntegrationSuiteBase extends JoinPushdownIntegrationSuiteBase`
   - `OracleJoinPushdownIntegrationSuiteBase extends JDBCJoinPushdownIntegrationSuiteBase`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213678585


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite

Review Comment:
   Sure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2213772589


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)

Review Comment:
   it might take the base method, so I changed it to `lazy` instead



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {

Review Comment:
   I would do it in separate PR if it's fine with you.



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  def namespaceOpt: Option[String] = None
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def qualifyTableName(tableName: String): String = namespaceOpt
+    .map(namespace => s"$namespace.$tableName").getOrElse(tableName)
+
+  private val fullyQualifiedTableName1: String = qualifyTableName(joinTableName1)
+
+  private val fullyQualifiedTableName2: String = qualifyTableName(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  protected def withConnection[T](f: Connection => T): T = {
+    val conn = DriverManager.getConnection(url, new Properties())
+    try {
+      f(conn)
+    } finally {
+      conn.close()
+    }
+  }
+
+  def dataPreparation(connection: Connection): Unit = {
+    schemaPreparation(connection)
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def schemaPreparation(connection: Connection): Unit = {
+    connection.prepareStatement(s"CREATE SCHEMA IF NOT EXISTS ${namespaceOpt.get}").executeUpdate()
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  AMOUNT ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  ADDRESS ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  NEXT_ID ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  SALARY ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  SURNAME ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin
+    ).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val address = s"address_$i"
+      (id, amount, address)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17)
+      val next_id = (id + 1) % 17
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, next_id, salary, surname)
+    }

Review Comment:
   42 is seed, so shouldn't be flaky



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "andrej-db (via GitHub)" <gi...@apache.org>.

andrej-db commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2212620811


##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala:
##########
@@ -233,7 +233,7 @@ private case class OracleDialect() extends JdbcDialect with SQLConfHelper with N
     extends JdbcSQLQueryBuilder(dialect, options) {
 
     override def build(): String = {
-      val selectStmt = s"SELECT $hintClause$columnList FROM ${options.tableOrQuery}" +
+      val selectStmt = s"SELECT $hintClause$columnList FROM $tableOrQuery" +

Review Comment:
   what if you supply a table via options?



##########
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/join/JDBCJoinPushdownIntegrationSuite.scala:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2.join
+
+import java.sql.Connection
+
+import org.apache.spark.sql.{DataFrame, QueryTest}
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.{DockerIntegrationFunSuite, JdbcDialect}
+import org.apache.spark.sql.jdbc.v2.V2JDBCPushdownTestUtils
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes}
+import org.apache.spark.tags.DockerTest
+
+@DockerTest
+trait JDBCJoinPushdownIntegrationSuite
+  extends QueryTest
+  with SharedSparkSession
+  with DockerIntegrationFunSuite
+  with V2JDBCPushdownTestUtils {
+  val catalogName: String
+  val namespaceOpt: Option[String] = None
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  // Concrete suite must provide the dialect for its DB
+  def jdbcDialect: JdbcDialect
+
+  private def catalogAndNamespace =
+    namespaceOpt.map(namespace => s"$catalogName.$namespace").getOrElse(catalogName)
+
+  def fullyQualifiedTableName1: String = namespaceOpt
+    .map(namespace => s"$namespace.$joinTableName1").getOrElse(joinTableName1)
+
+  def fullyQualifiedTableName2: String = namespaceOpt
+    .map(namespace => s"$namespace.$joinTableName2").getOrElse(joinTableName2)
+
+  protected def getJDBCTypeString(dt: DataType): String = {
+    JdbcUtils.getJdbcType(dt, jdbcDialect).databaseTypeDefinition.toUpperCase()
+  }
+
+  protected def caseConvert(tableName: String): String = tableName
+
+  def dataPreparation(connection: Connection): Unit = {
+    tablePreparation(connection)
+    fillJoinTables(connection)
+  }
+
+  def tablePreparation(connection: Connection): Unit = {
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName1 (
+         |  id ${getJDBCTypeString(DataTypes.IntegerType)},
+         |  amount ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  name ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin).executeUpdate()
+
+    connection.prepareStatement(
+      s"""CREATE TABLE $fullyQualifiedTableName2 (
+         |  id ${getJDBCTypeString(DataTypes.LongType)},
+         |  salary ${getJDBCTypeString(DataTypes.createDecimalType(10, 2))},
+         |  surname ${getJDBCTypeString(DataTypes.StringType)}
+         |)""".stripMargin).executeUpdate()
+  }
+
+  def fillJoinTables(connection: Connection): Unit = {
+    val random = new java.util.Random(42)
+    val table1Data = (1 to 100).map { i =>
+      val id = i % 11
+      val amount = BigDecimal.valueOf(random.nextDouble() * 10000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val name = s"name_$i"
+      (id, amount, name)
+    }
+    val table2Data = (1 to 100).map { i =>
+      val id = (i % 17).toLong
+      val salary = BigDecimal.valueOf(random.nextDouble() * 50000)
+        .setScale(2, BigDecimal.RoundingMode.HALF_UP)
+      val surname = s"surname_$i"
+      (id, salary, surname)
+    }
+
+    // Use parameterized queries to handle different data types properly
+    val insertStmt1 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName1 (id, amount, name) VALUES (?, ?, ?)"
+    )
+    table1Data.foreach { case (id, amount, name) =>
+      insertStmt1.setInt(1, id)
+      insertStmt1.setBigDecimal(2, amount.bigDecimal)
+      insertStmt1.setString(3, name)
+      insertStmt1.executeUpdate()
+    }
+    insertStmt1.close()
+
+    val insertStmt2 = connection.prepareStatement(
+      s"INSERT INTO $fullyQualifiedTableName2 (id, salary, surname) VALUES (?, ?, ?)"
+    )
+    table2Data.foreach { case (id, salary, surname) =>
+      insertStmt2.setLong(1, id)
+      insertStmt2.setBigDecimal(2, salary.bigDecimal)
+      insertStmt2.setString(3, surname)
+      insertStmt2.executeUpdate()
+    }
+    insertStmt2.close()
+  }
+
+  /**
+   * Runs the plan and makes sure the plans contains all of the keywords.
+   */
+  protected def checkKeywordsExistsInExplain(df: DataFrame, keywords: String*): Unit = {
+    val output = new java.io.ByteArrayOutputStream()
+    Console.withOut(output) {
+      df.explain(extended = true)
+    }
+    val normalizedOutput = output.toString.replaceAll("#\\d+", "#x")
+    for (key <- keywords) {
+      assert(normalizedOutput.contains(key), s"Expected keyword '$key' not found in explain output")
+    }
+  }
+
+  private def checkPushedInfo(df: DataFrame, expectedPlanFragment: String*): Unit = {
+    withSQLConf(SQLConf.MAX_METADATA_STRING_LENGTH.key -> "1000") {
+      df.queryExecution.optimizedPlan.collect {
+        case _: DataSourceV2ScanRelation =>
+          checkKeywordsExistsInExplain(df, expectedPlanFragment: _*)
+      }
+    }
+  }
+
+  private def checkJoinNotPushed(df: DataFrame): Unit = {
+    val joinNodes = df.queryExecution.optimizedPlan.collect {
+      case j: Join => j
+    }
+    assert(joinNodes.nonEmpty, "Join should not be pushed down")
+  }
+
+  private def checkJoinPushed(df: DataFrame, expectedTables: String*): Unit = {
+    val joinNodes = df.queryExecution.optimizedPlan.collect {
+      case j: Join => j
+    }
+    assert(joinNodes.isEmpty, "Join should be pushed down")
+    if (expectedTables.nonEmpty) {
+      checkPushedInfo(df, s"PushedJoins: [${expectedTables.mkString(", ")}]")
+    }
+  }
+
+  test("Test basic inner join pushdown with column pruning") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount, t2.salary
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    // Verify we have non-empty results
+    assert(rows.nonEmpty, "Join should produce non-empty results")
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+
+  test("Test join with additional filters") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount, t2.salary
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |WHERE t1.amount > 5000 AND t2.salary > 25000
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkFilterPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test self join should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.name as name2
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName1)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join without condition should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |CROSS JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with complex condition") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname, t1.amount + t2.salary as total
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2
+      |ON t1.id = t2.id AND t1.amount > 1000
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with limit and order") {
+    // ORDER BY is used to have same ordering on Spark and database. Otherwise, different results
+    // could be returned.
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |ORDER BY t1.id, t1.name, t2.surname
+      |LIMIT 5
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkSortRemoved(df)
+      checkLimitRemoved(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with order by") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |ORDER BY t1.id, t1.name, t2.surname
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      // Order without limit is not supported in DSv2
+      checkSortRemoved(df, false)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with aggregation") {
+    val sqlQuery = s"""
+      |SELECT t1.id, COUNT(*), AVG(t1.amount), MAX(t2.salary)
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |GROUP BY t1.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAggregateRemoved(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test left outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |LEFT JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test right outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |RIGHT JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test full outer join should not be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |FULL OUTER JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id = t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinNotPushed(df)
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with subquery should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, sub.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN (
+      |  SELECT id, surname FROM $catalogAndNamespace.${caseConvert(joinTableName2)}
+      |  WHERE salary > 25000
+      |) sub ON t1.id = sub.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+
+  test("Test join with non-equality condition should be pushed down") {
+    val sqlQuery = s"""
+      |SELECT t1.id, t1.name, t2.surname
+      |FROM $catalogAndNamespace.${caseConvert(joinTableName1)} t1
+      |JOIN $catalogAndNamespace.${caseConvert(joinTableName2)} t2 ON t1.id > t2.id
+      |""".stripMargin
+
+    val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") {
+      sql(sqlQuery).collect().toSeq
+    }
+
+    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
+      val df = sql(sqlQuery)
+      checkJoinPushed(
+        df,
+        s"$catalogAndNamespace.${caseConvert(joinTableName1)}",
+        s"$catalogAndNamespace.${caseConvert(joinTableName2)}"
+      )
+      checkAnswer(df, rows)
+    }
+  }
+}

Review Comment:
   can we have an ANTI JOIN as well?
   or is that unsupported?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2214799545


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")

Review Comment:
   why it's an opt? most dialects much have a schema, right?



##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"
+  override val namespaceOpt: Option[String] = Some("test")

Review Comment:
   why it's an opt? most dialects must have a schema, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "PetarVasiljevic-DB (via GitHub)" <gi...@apache.org>.

PetarVasiljevic-DB commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2215311487


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownSuite.scala:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.Connection
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{ExplainSuiteHelper, QueryTest}
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{H2Dialect, JdbcDialect}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.Utils
+
+class JDBCV2JoinPushdownSuite
+  extends QueryTest
+  with SharedSparkSession
+  with ExplainSuiteHelper
+  with DataSourcePushdownTestUtils
+  with JDBCV2JoinPushdownIntegrationSuiteBase {
+  val tempDir = Utils.createTempDir()
+  override val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+  override val catalogName: String = "h2"

Review Comment:
   As above, followed the pattern in `V2JDBCTest`. Changed it to `join_pushdown_catalog` in common trait



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on code in PR #51519:
URL: https://github.com/apache/spark/pull/51519#discussion_r2216280568


##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala:
##########
@@ -0,0 +1,578 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc.v2
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.connector.DataSourcePushdownTestUtils
+import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.jdbc.JdbcDialect
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, DataTypes, StructField, StructType}
+
+trait JDBCV2JoinPushdownIntegrationSuiteBase
+  extends QueryTest
+  with SharedSparkSession
+  with DataSourcePushdownTestUtils {
+  val catalogName: String = "join_pushdown_catalog"
+  val namespace: String = "join_schema"
+  val url: String
+
+  val joinTableName1: String = "join_table_1"
+  val joinTableName2: String = "join_table_2"
+
+  val jdbcDialect: JdbcDialect
+
+  override def sparkConf: SparkConf = super.sparkConf
+    .set(s"spark.sql.catalog.$catalogName", classOf[JDBCTableCatalog].getName)
+    .set(s"spark.sql.catalog.$catalogName.url", url)
+    .set(s"spark.sql.catalog.$catalogName.pushDownJoin", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownAggregate", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownLimit", "true")
+    .set(s"spark.sql.catalog.$catalogName.pushDownOffset", "true")
+
+  private def catalogAndNamespace = s"$catalogName.$namespace"
+
+  def qualifyTableName(tableName: String): String = s"$namespace.$tableName"

Review Comment:
   since we have `jdbcDialect`, shall we just call it to qualify the names?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org