You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/16 05:43:23 UTC
[GitHub] [spark] beliefer opened a new pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
beliefer opened a new pull request #34918:
URL: https://github.com/apache/spark/pull/34918
### What changes were proposed in this pull request?
Currently, Spark supports push down limit to data source.
However, in the user's scenario, limit must have the premise of order by. Because limit and order by are more valuable together.
On the other hand, push down top N(same as order by ... limit N) outputs the data with basic order to Spark sort, the the sort of Spark may have some performance improvement.
### Why are the changes needed?
1. push down top N is very useful for users scenario.
2. push down top N could improves the performance of sort.
### Does this PR introduce _any_ user-facing change?
'No'. Just change the physical execute.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995477394
**[Test build #146262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146262/testReport)** for PR 34918 at commit [`de651c2`](https://github.com/apache/spark/commit/de651c2d8cd9477f4c993ca8812d650d703a392a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34918:
URL: https://github.com/apache/spark/pull/34918#discussion_r770775802
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -726,6 +726,24 @@ object DataSourceStrategy
}
}
+ protected[sql] def translateSortOrders(sortOrders: Seq[SortOrder]): Seq[SortOrderV2] = {
+ def translateOortOrder(sortOrder: SortOrder): Option[SortOrderV2] = sortOrder match {
+ case SortOrder(PushableColumnWithoutNestedColumn(name), directionV1, nullOrderingV1, _) =>
+ val directionV2 = directionV1 match {
+ case Ascending => SortDirection.ASCENDING
+ case Descending => SortDirection.DESCENDING
+ }
+ val nullOrderingV2 = nullOrderingV1 match {
+ case NullsFirst => NullOrdering.NULLS_FIRST
+ case NullsLast => NullOrdering.NULLS_LAST
+ }
+ Some(SortValue(FieldReference(name), directionV2, nullOrderingV2))
+ case _ => None
+ }
+
+ sortOrders.flatMap(translateOortOrder)
Review comment:
Thank you, @beliefer !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995767979
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146287/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995636945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50743/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995865005
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50760/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995865005
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50760/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34918:
URL: https://github.com/apache/spark/pull/34918#discussion_r770300655
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -726,6 +726,24 @@ object DataSourceStrategy
}
}
+ protected[sql] def translateSortOrders(sortOrders: Seq[SortOrder]): Seq[SortOrderV2] = {
+ def translateOortOrder(sortOrder: SortOrder): Option[SortOrderV2] = sortOrder match {
+ case SortOrder(PushableColumnWithoutNestedColumn(name), directionV1, nullOrderingV1, _) =>
+ val directionV2 = directionV1 match {
+ case Ascending => SortDirection.ASCENDING
+ case Descending => SortDirection.DESCENDING
+ }
+ val nullOrderingV2 = nullOrderingV1 match {
+ case NullsFirst => NullOrdering.NULLS_FIRST
+ case NullsLast => NullOrdering.NULLS_LAST
+ }
+ Some(SortValue(FieldReference(name), directionV2, nullOrderingV2))
+ case _ => None
+ }
+
+ sortOrders.flatMap(translateOortOrder)
Review comment:
Let's do the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995547024
**[Test build #146268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146268/testReport)** for PR 34918 at commit [`37902c0`](https://github.com/apache/spark/commit/37902c0bb742d395911790619251c6569447c72d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995841552
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50760/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995771087
**[Test build #146288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146288/testReport)** for PR 34918 at commit [`bbecf9d`](https://github.com/apache/spark/commit/bbecf9d956db74c0815f7fe6928befd17d31893c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-996021700
**[Test build #146288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146288/testReport)** for PR 34918 at commit [`bbecf9d`](https://github.com/apache/spark/commit/bbecf9d956db74c0815f7fe6928befd17d31893c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995477394
**[Test build #146262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146262/testReport)** for PR 34918 at commit [`de651c2`](https://github.com/apache/spark/commit/de651c2d8cd9477f4c993ca8812d650d703a392a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995635429
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50743/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34918:
URL: https://github.com/apache/spark/pull/34918#discussion_r770255687
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -726,6 +726,24 @@ object DataSourceStrategy
}
}
+ protected[sql] def translateSortOrders(sortOrders: Seq[SortOrder]): Seq[SortOrderV2] = {
+ def translateOortOrder(sortOrder: SortOrder): Option[SortOrderV2] = sortOrder match {
+ case SortOrder(PushableColumnWithoutNestedColumn(name), directionV1, nullOrderingV1, _) =>
+ val directionV2 = directionV1 match {
+ case Ascending => SortDirection.ASCENDING
+ case Descending => SortDirection.DESCENDING
+ }
+ val nullOrderingV2 = nullOrderingV1 match {
+ case NullsFirst => NullOrdering.NULLS_FIRST
+ case NullsLast => NullOrdering.NULLS_LAST
+ }
+ Some(SortValue(FieldReference(name), directionV2, nullOrderingV2))
+ case _ => None
+ }
+
+ sortOrders.flatMap(translateOortOrder)
Review comment:
The origin PR https://github.com/apache/spark/pull/34738 issues(ref https://github.com/apache/spark/pull/34738#discussion_r770002006) an incompatible error when building with Scala 2.13.
I updated the code in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995716682
**[Test build #146287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146287/testReport)** for PR 34918 at commit [`31d4ea5`](https://github.com/apache/spark/commit/31d4ea52ed5ba9e50cfd29a66799e758da9effc3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995544690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50737/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995802925
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50760/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-996024666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995716682
**[Test build #146287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146287/testReport)** for PR 34918 at commit [`31d4ea5`](https://github.com/apache/spark/commit/31d4ea52ed5ba9e50cfd29a66799e758da9effc3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995868278
Github action passed, I'm merging it to master, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995706474
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995636945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50743/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995721345
**[Test build #146287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146287/testReport)** for PR 34918 at commit [`31d4ea5`](https://github.com/apache/spark/commit/31d4ea52ed5ba9e50cfd29a66799e758da9effc3).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995875410
@cloud-fan Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34918:
URL: https://github.com/apache/spark/pull/34918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995706474
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995582970
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50743/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34918:
URL: https://github.com/apache/spark/pull/34918#discussion_r770253218
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -726,6 +726,24 @@ object DataSourceStrategy
}
}
+ protected[sql] def translateSortOrders(sortOrders: Seq[SortOrder]): Seq[SortOrderV2] = {
+ def translateOortOrder(sortOrder: SortOrder): Option[SortOrderV2] = sortOrder match {
+ case SortOrder(PushableColumnWithoutNestedColumn(name), directionV1, nullOrderingV1, _) =>
+ val directionV2 = directionV1 match {
+ case Ascending => SortDirection.ASCENDING
+ case Descending => SortDirection.DESCENDING
+ }
+ val nullOrderingV2 = nullOrderingV1 match {
+ case NullsFirst => NullOrdering.NULLS_FIRST
+ case NullsLast => NullOrdering.NULLS_LAST
+ }
+ Some(SortValue(FieldReference(name), directionV2, nullOrderingV2))
+ case _ => None
+ }
+
+ sortOrders.flatMap(translateOortOrder)
Review comment:
is it safe to return a partial sort order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995504485
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50737/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995544690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50737/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995704045
**[Test build #146262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146262/testReport)** for PR 34918 at commit [`de651c2`](https://github.com/apache/spark/commit/de651c2d8cd9477f4c993ca8812d650d703a392a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995767979
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146287/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995813451
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146268/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995547024
**[Test build #146268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146268/testReport)** for PR 34918 at commit [`37902c0`](https://github.com/apache/spark/commit/37902c0bb742d395911790619251c6569447c72d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995806830
**[Test build #146268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146268/testReport)** for PR 34918 at commit [`37902c0`](https://github.com/apache/spark/commit/37902c0bb742d395911790619251c6569447c72d).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995531738
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50737/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995771087
**[Test build #146288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146288/testReport)** for PR 34918 at commit [`bbecf9d`](https://github.com/apache/spark/commit/bbecf9d956db74c0815f7fe6928befd17d31893c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-996024666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34918: [SPARK-37483][SQL] Support push down top N to JDBC data source V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34918:
URL: https://github.com/apache/spark/pull/34918#issuecomment-995813451
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146268/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org