You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hu...@apache.org on 2021/11/03 21:36:59 UTC
[spark] branch branch-3.2 updated: [MINOR][SQL][DOCS] Document JDBC
aggregate push down is for DSV2 only
This is an automated email from the ASF dual-hosted git repository.
huaxingao pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 8804ed9 [MINOR][SQL][DOCS] Document JDBC aggregate push down is for DSV2 only
8804ed9 is described below
commit 8804ed9536f5e1bb3a24963086e8b093a0f71d2f
Author: Huaxin Gao <hu...@apple.com>
AuthorDate: Wed Nov 3 14:34:33 2021 -0700
[MINOR][SQL][DOCS] Document JDBC aggregate push down is for DSV2 only
### What changes were proposed in this pull request?
To specify JDBC aggregate push down is for DS V2 only. This change is for both 3.2 and master.
### Why are the changes needed?
To make the doc clear so user won't use aggregate push down in DS v1.
### Does this PR introduce _any_ user-facing change?
No. Doc change only
### How was this patch tested?
Manually checked.
Closes #34465 from huaxingao/doc_minor.
Authored-by: Huaxin Gao <hu...@apple.com>
Signed-off-by: Huaxin Gao <hu...@apple.com>
(cherry picked from commit 44151be395852a5ef7e5784607f4e69da5e61569)
Signed-off-by: Huaxin Gao <hu...@apple.com>
---
docs/sql-data-sources-jdbc.md | 2 +-
.../org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/sql-data-sources-jdbc.md b/docs/sql-data-sources-jdbc.md
index 315f476..4d38923 100644
--- a/docs/sql-data-sources-jdbc.md
+++ b/docs/sql-data-sources-jdbc.md
@@ -241,7 +241,7 @@ logging into the data sources.
<td><code>pushDownAggregate</code></td>
<td><code>false</code></td>
<td>
- The option to enable or disable aggregate push-down into the JDBC data source. The default value is false, in which case Spark will not push down aggregates to the JDBC data source. Otherwise, if sets to true, aggregates will be pushed down to the JDBC data source. Aggregate push-down is usually turned off when the aggregate is performed faster by Spark than by the JDBC data source. Please note that aggregates can be pushed down if and only if all the aggregate functions and the rel [...]
+ The option to enable or disable aggregate push-down in V2 JDBC data source. The default value is false, in which case Spark will not push down aggregates to the JDBC data source. Otherwise, if sets to true, aggregates will be pushed down to the JDBC data source. Aggregate push-down is usually turned off when the aggregate is performed faster by Spark than by the JDBC data source. Please note that aggregates can be pushed down if and only if all the aggregate functions and the relate [...]
</td>
<td>read</td>
</tr>
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
index 8b2ae2b..43d0481 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
@@ -189,6 +189,7 @@ class JDBCOptions(
val pushDownPredicate = parameters.getOrElse(JDBC_PUSHDOWN_PREDICATE, "true").toBoolean
// An option to allow/disallow pushing down aggregate into JDBC data source
+ // This only applies to Data Source V2 JDBC
val pushDownAggregate = parameters.getOrElse(JDBC_PUSHDOWN_AGGREGATE, "false").toBoolean
// The local path of user's keytab file, which is assumed to be pre-uploaded to all nodes either
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org