You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/06 09:34:07 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
AngersZhuuuu opened a new pull request #30278:
URL: https://github.com/apache/spark/pull/30278
### What changes were proposed in this pull request?
Push down filter throw expand. For case below:
```
create table SPARK_33302_1(pid int, uid int, sid int, dt date, suid int) using parquet;
create table SPARK_33302_2(pid int, vs int, uid int, csid int) using parquet;
SELECT
years,
appversion,
SUM(uusers) AS users
FROM (SELECT
Date_trunc('year', dt) AS years,
CASE
WHEN h.pid = 3 THEN 'iOS'
WHEN h.pid = 4 THEN 'Android'
ELSE 'Other'
END AS viewport,
h.vs AS appversion,
Count(DISTINCT u.uid) AS uusers
,Count(DISTINCT u.suid) AS srcusers
FROM SPARK_33302_1 u
join SPARK_33302_2 h
ON h.uid = u.uid
GROUP BY 1,
2,
3) AS a
WHERE viewport = 'iOS'
GROUP BY 1,
2
```
Plan. before this pr:
```
== Physical Plan ==
*(5) HashAggregate(keys=[years#30, appversion#32], functions=[sum(uusers#33L)])
+- Exchange hashpartitioning(years#30, appversion#32, 200), true, [id=#251]
+- *(4) HashAggregate(keys=[years#30, appversion#32], functions=[partial_sum(uusers#33L)])
+- *(4) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[count(if ((gid#44 = 1)) u.`uid`#47 else null)])
+- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, 200), true, [id=#246]
+- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[partial_count(if ((gid#44 = 1)) u.`uid`#47 else null)])
+- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
+- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44, 200), true, [id=#241]
+- *(2) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
+- *(2) Filter (CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46 = iOS)
+- *(2) Expand [ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, uid#7, null, 1), ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, null, suid#10, 2)], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44]
+- *(2) Project [uid#7, dt#9, suid#10, pid#11, vs#12]
+- *(2) BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight
:- *(2) Project [uid#7, dt#9, suid#10]
: +- *(2) Filter isnotnull(uid#7)
: +- *(2) ColumnarToRow
: +- FileScan parquet default.spark_33301_1[uid#7,dt#9,suid#10] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/spark_33301_1], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date,suid:int>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, true] as bigint))), [id=#233]
+- *(1) Project [pid#11, vs#12, uid#13]
+- *(1) Filter isnotnull(uid#13)
+- *(1) ColumnarToRow
+- FileScan parquet default.spark_33301_2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [isnotnull(uid#13)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/spark_33301_2], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>
```
Plan. after. this pr. :
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[years#0, appversion#2], functions=[sum(uusers#3L)], output=[years#0, appversion#2, users#5L])
+- Exchange hashpartitioning(years#0, appversion#2, 5), true, [id=#71]
+- HashAggregate(keys=[years#0, appversion#2], functions=[partial_sum(uusers#3L)], output=[years#0, appversion#2, sum#22L])
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[count(distinct uid#7)], output=[years#0, appversion#2, uusers#3L])
+- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, 5), true, [id=#67]
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[partial_count(distinct uid#7)], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, count#27L])
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
+- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7, 5), true, [id=#63]
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles)) AS date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END AS CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
+- Project [uid#7, dt#9, pid#11, vs#12]
+- BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight, false
:- Filter isnotnull(uid#7)
: +- FileScan parquet default.t1[uid#7,dt#9] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, false] as bigint)),false), [id=#58]
+- Filter ((CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS) AND isnotnull(uid#13))
+- FileScan parquet default.t2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [(CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS), isnotnull..., Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>
```
### Why are the changes needed?
Improve performance, filter more data.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519625559
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,36 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
+ test("push down predicate through expand") {
+ val input = LocalRelation('a.int, 'b.string, 'c.double)
+ val query =
+ Aggregate(
Review comment:
why does this test need an Aggregate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724717554
**[Test build #130850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130850/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724373354
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723030808
**[Test build #130723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130723/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723038581
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35324/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724580268
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35458/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724534408
**[Test build #130832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130832/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724161430
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724464732
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724535194
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130832/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724538020
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724744511
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724161430
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r520195535
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
+ test("push down predicate through expand") {
+ val input = LocalRelation('a.int, 'b.string, 'c.double)
Review comment:
> nit: could you use `testRelation` instead?
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724034504
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723078713
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519668085
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> how does it related to left semi join?
Not related, I didn't notice that the API would be used by `PushDownLeftSemiAntiJoin `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723038551
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35324/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723162801
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519659900
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> > seems current master fix this case
>
> What do you mean by "fix this case"?
I have found the pr #29673
Before this pr, SQL
```
SELECT
years,
appversion,
SUM(uusers) AS users
FROM (SELECT
Date_trunc('year', dt) AS years,
CASE
WHEN h.pid = 3 THEN 'iOS'
WHEN h.pid = 4 THEN 'Android'
ELSE 'Other'
END AS viewport,
h.vs AS appversion,
Count(DISTINCT u.uid) AS uusers
,Count(DISTINCT u.suid) AS srcusers
FROM t1 u
join t2 h
ON h.uid = u.uid
GROUP BY 1,
2,
3) AS a
WHERE viewport = 'iOS'
GROUP BY 1,
2
```
Optimized plan is
```
== Optimized Logical Plan ==
Aggregate [years#0, appversion#2], [years#0, appversion#2, sum(uusers#3L) AS users#5L]
+- Aggregate [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24 AS years#0, vs#17 AS appversion#2, count(if ((gid#23 = 1)) u.`uid`#26 else null) AS uusers#3L]
+- Aggregate [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, u.`suid`#27, gid#23], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, gid#23]
+- Filter (CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25 = iOS)
+- Expand [ArrayBuffer(date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17, uid#12, null, 1), ArrayBuffer(date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17, null, suid#15, 2)], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, u.`suid`#27, gid#23]
+- Project [uid#12, dt#14, suid#15, pid#16, vs#17]
+- Join Inner, (uid#18 = uid#12)
:- Project [uid#12, dt#14, suid#15]
: +- Filter isnotnull(uid#12)
: +- Relation[pid#11,uid#12,sid#13,dt#14,suid#15] parquet
+- Project [pid#16, vs#17, uid#18]
+- Filter isnotnull(uid#18)
+- Relation[pid#16,vs#17,uid#18,csid#19] parquet
```
After that pr, Optimized plan is
```
== Optimized Logical Plan ==
Aggregate [years#0, appversion#2], [years#0, appversion#2, sum(uusers#3L) AS users#5L]
+- Aggregate [date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17], [date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)) AS years#0, vs#17 AS appversion#2, count(distinct uid#12) AS uusers#3L]
+- Project [uid#12, dt#14, pid#16, vs#17]
+- Join Inner, (uid#18 = uid#12)
:- Project [uid#12, dt#14]
: +- Filter isnotnull(uid#12)
: +- Relation[pid#11,uid#12,sid#13,dt#14,suid#15] parquet
+- Project [pid#16, vs#17, uid#18]
+- Filter ((CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END = iOS) AND isnotnull(uid#18))
+- Relation[pid#16,vs#17,uid#18,csid#19] parquet
```
`Filter((CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END = iOS))` is pushed down and won't generate `Expand`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724591979
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35458/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519924222
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> The optimized (left-side) plan above looks correct to me...
My fault, I misunderstand some code about `PushDownLeftSemiAntiJoin`, test case added ==
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723078713
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724354059
> > Push down filter throw expand. For case below:
>
> typo? `throw` -> `through`
Yea, updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724447421
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723030141
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519503774
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
This change affects the `PushDownLeftSemiAntiJoin` rule, too. So, could you add tests for the case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724034504
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724034485
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35395/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724592008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724473903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30278:
URL: https://github.com/apache/spark/pull/30278
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724448899
**[Test build #130832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130832/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724448899
**[Test build #130832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130832/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r520190103
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
+ test("push down predicate through expand") {
+ val input = LocalRelation('a.int, 'b.string, 'c.double)
Review comment:
nit: could you use `testRelation` instead?
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
Review comment:
nit: unnecessary blank.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723162801
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724720527
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-722979708
**[Test build #130714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130714/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724383358
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35418/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724260824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724383353
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724151596
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724447123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519674067
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> This change affects the PushDownLeftSemiAntiJoin rule, too. So, could you add tests for the case?
So can we add such a test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724259862
**[Test build #130797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130797/testReport)** for PR 30278 at commit [`77d6e45`](https://github.com/apache/spark/commit/77d6e45ceb7b21dcc8fabf05b9b40102bd0b645b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724473883
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723999420
**[Test build #130787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130787/testReport)** for PR 30278 at commit [`803bf0a`](https://github.com/apache/spark/commit/803bf0a6ac5f68e634fbc0980c75342ffee3f51a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724137339
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35406/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r520195621
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
Review comment:
> nit: unnecessary blank.
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723024488
**[Test build #130714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130714/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723024805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130714/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519923126
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> I'm sorry I didn't get it. What's the issue here? We can't pushdown left-semi join through expand?
oh..my mistake, I misunderstood some code about `PushDownLeftSemiAntiJoin `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723024800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724383339
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724473903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519918205
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> I'm sorry I didn't get it. What's the issue here? We can't pushdown left-semi join through expand?
I just want to show, since we can push down left-semi join through expand, so we won't meet case impacted by `canPushThrough` after add `case _: Expand => true` in `PushDownLeftSemiAntiJoin ` . So we can't write a test about https://github.com/apache/spark/pull/30278/files#r519674067 to show the influence about. comment https://github.com/apache/spark/pull/30278#discussion_r519503774.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519786610
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
##########
@@ -1208,6 +1208,36 @@ class FilterPushdownSuite extends PlanTest {
checkAnalysis = false)
}
+
+ test("push down predicate through expand") {
+ val input = LocalRelation('a.int, 'b.string, 'c.double)
+ val query =
+ Aggregate(
Review comment:
> why does this test need an Aggregate?
Not necessary, remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519538842
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> This change affects the `PushDownLeftSemiAntiJoin` rule, too. So, could you add tests for the case?
Double check the case, seems current master fix this case by some pr, but 3.0 is still as jira desc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724540303
**[Test build #130850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130850/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724446391
**[Test build #130808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130808/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519627065
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> seems current master fix this case
What do you mean by "fix this case"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724151574
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35406/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723038573
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724160172
**[Test build #130787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130787/testReport)** for PR 30278 at commit [`803bf0a`](https://github.com/apache/spark/commit/803bf0a6ac5f68e634fbc0980c75342ffee3f51a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724108474
**[Test build #130797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130797/testReport)** for PR 30278 at commit [`77d6e45`](https://github.com/apache/spark/commit/77d6e45ceb7b21dcc8fabf05b9b40102bd0b645b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724350167
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723065793
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35332/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724540303
**[Test build #130850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130850/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723161769
**[Test build #130723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130723/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724022383
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35395/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723029109
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35324/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724260824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724535187
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724383353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723078695
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35332/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724720527
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519836853
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
The optimized (left-side) plan above looks correct to me...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519696759
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
> > This change affects the PushDownLeftSemiAntiJoin rule, too. So, could you add tests for the case?
>
> So can we add such a test?
With test case in `LeftSemiPushdownSuite`
```
test("Unary: LeftSemi join push down through expand") {
val expand = Expand(Seq(Seq('a, 'b, "null"), Seq('a, "null", 'c)),
Seq('a, 'b, 'c), testRelation)
val originalQuery = expand
.join(testRelation1, joinType = LeftSemi, condition = Some('b === 'd && 'b === 1))
val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer = Expand(Seq(Seq('a, 'b, "null"), Seq('a, "null", 'c)),
Seq('a, 'b, 'c), Filter(EqualTo('b, 1), testRelation))
.join(testRelation1, joinType = LeftSemi, condition = Some('b === 'd))
.analyze
comparePlans(optimized, correctAnswer)
}
```
originalQuery is
```
'Join LeftSemi, (('b = 'd) AND ('b = 1))
:- 'Expand [List('a, 'b, null), List('a, null, 'c)], ['a, 'b, 'c]
: +- LocalRelation <empty>, [a#0, b#1, c#2]
+- LocalRelation <empty>, [d#3]
```
Test result is
```
== FAIL: Plans do not match ===
!'Expand [List(a#0, b#0, null), List(a#0, null, c#0)], [a#0, b#0, c#0] 'Join LeftSemi, (b#0 = d#0)
!+- 'Join LeftSemi, ((b#0 = 1) AND (b#0 = d#0)) :- Expand [List(a#0, b#0, null), List(a#0, null, c#0)], [a#0, b#0, c#0]
! :- LocalRelation <empty>, [a#0, b#0, c#0] : +- Filter (b#0 = 1)
! +- LocalRelation <empty>, [d#0] : +- LocalRelation <empty>, [a#0, b#0, c#0]
! +- LocalRelation <empty>, [d#0]
```
Expand will be promoted below Join, so should we ignore this case or add a parameter in `canPushThrough` like below
```
def canPushThrough(p: UnaryNode, isFilterPushDown: Boolean = false): Boolean = p match {
// Note that some operators (e.g. project, aggregate, union) are being handled separately
// (earlier in this rule).
case _: AppendColumns => true
case _: Distinct => true
case _: Generate => true
case _: Pivot => true
case _: RepartitionByExpression => true
case _: Repartition => true
case _: ScriptTransformation => true
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
case _: Expand => isFilterPushDown
case _ => false
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724108474
**[Test build #130797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130797/testReport)** for PR 30278 at commit [`77d6e45`](https://github.com/apache/spark/commit/77d6e45ceb7b21dcc8fabf05b9b40102bd0b645b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723577483
gentle ping @maropu @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519666883
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
how does it related to left semi join?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724447123
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-722979708
**[Test build #130714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130714/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724151609
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35406/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723024800
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30278:
URL: https://github.com/apache/spark/pull/30278#discussion_r519789694
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
case _: Sort => true
case _: BatchEvalPython => true
case _: ArrowEvalPython => true
+ case _: Expand => true
Review comment:
I'm sorry I didn't get it. What's the issue here? We can't pushdown left-semi join through expand?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724447131
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130808/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723038573
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723999420
**[Test build #130787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130787/testReport)** for PR 30278 at commit [`803bf0a`](https://github.com/apache/spark/commit/803bf0a6ac5f68e634fbc0980c75342ffee3f51a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724356732
**[Test build #130808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130808/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724356732
**[Test build #130808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130808/testReport)** for PR 30278 at commit [`a09f836`](https://github.com/apache/spark/commit/a09f8364cf4b9853cdf1ff4dc00d761b3a5b6291).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-723030808
**[Test build #130723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130723/testReport)** for PR 30278 at commit [`c3ec848`](https://github.com/apache/spark/commit/c3ec8488d945712d6e2a19987e205f5aee5146de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724151596
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724592008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30278: [SPARK-33302][SQL] Push down filters through Expand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30278:
URL: https://github.com/apache/spark/pull/30278#issuecomment-724535187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org