You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gao8658 <gi...@git.apache.org> on 2014/06/19 06:41:12 UTC
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
GitHub user gao8658 opened a pull request:
https://github.com/apache/spark/pull/1127
Spark SQL add LeftSemiBloomFilterBroadcastJoin
Hi ,All .
I want to submit a join operator called
LeftSemiBloomFilterBroadcastJoin (LeftSemiJoinBFB)
Sometimes the Semijoin's broadcast table can't fit memory.So we can make it as Bloomfilter to reduce the space and then broadcast it do the mapside join .
Some code reference HashJoin and BroadcastNestedLoopJoin implementation.
The bloomfilter code use Shark's BloomFilter class implementation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gao8658/spark patch-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1127.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1127
----
commit 470243cce24bee9a6c49a2f1538932ff3dba8f64
Author: Yanjie Gao <ga...@163.com>
Date: 2014-06-19T04:40:53Z
Spark SQL add LeftSemiBloomFilterBroadcastJoin
Hi ,All .
I want to submit a join operator called
LeftSemiBloomFilterBroadcastJoin (LeftSemiJoinBFB)
Sometimes the Semijoin's broadcast table can't fit memory.So we can make it as Bloomfilter to reduce the space and then broadcast it do the mapside join .
Some code reference HashJoin and BroadcastNestedLoopJoin implementation.
The bloomfilter code use Shark's BloomFilter class implementation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by gao8658 <gi...@git.apache.org>.
Github user gao8658 commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-46636649
Thanks a lot ,I will reformat it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by concretevitamin <gi...@git.apache.org>.
Github user concretevitamin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1127#discussion_r13989753
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala ---
@@ -245,6 +245,74 @@ case class LeftSemiJoinBNL(
}
}
+
+
+
+/*
+LeftSemiBloomFilterBroadcastJoin
+Sometimes the semijoin's broadcast table can't fit memory.So we can make it as Bloomfilter to reduce the space
+and then broadcast it do the mapside join
+THe bloomfilter use Shark's BloomFilter class implementation.
+*/
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class LeftSemiJoinBFB(
+ leftKeys: Seq[Expression],
--- End diff --
4 space indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...
Posted by YanjieGao <gi...@git.apache.org>.
Github user YanjieGao closed the pull request at:
https://github.com/apache/spark/pull/1127
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-54251469
Hi @YanjieGao, thank for working on this! I think it would be great to support this optimization. However, I think the hardest part here is going to be figuring how to hook this into the planner such that it is chosen when the data requires it, and otherwise we use the standard join algorithms. Since I think that is going to a pretty large task, perhaps it would be best to close this issue for now and revisit it when we have a full design for how choose join operators.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1127#discussion_r14051069
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala ---
@@ -245,6 +245,73 @@ case class LeftSemiJoinBNL(
}
}
+
+
+
+
+/**
+ * :: DeveloperApi ::
+ * LeftSemiBloomFilterBroadcastJoin
+ * Sometimes the semijoin's broadcast table can't fit memory.So we can make it as Bloomfilter to reduce the space
+ * and then broadcast it do the mapside join
+ * The bloomfilter use Shark's BloomFilter class implementation.
+ */
+@DeveloperApi
+case class LeftSemiJoinBFB(
+ leftKeys: Seq[Expression],
--- End diff --
Indent 4 spaces. Also I'd go with the full more descriptive name instead of BFB since we are only going to have to type it out in like 2 places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...
Posted by YanjieGao <gi...@git.apache.org>.
Github user YanjieGao commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-54283611
Hi marmbrus , Got it , if i have some other good idea i will try to communicate with you ,Thanks ,I will close it latter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by YanjieGao <gi...@git.apache.org>.
Github user YanjieGao commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-46754487
Hi Zongheng, I reformat the code .I don't know if that is ok. And i hope you can give me more suggestions . Thanks a lot
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by YanjieGao <gi...@git.apache.org>.
Github user YanjieGao commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-46769416
Thanks a lot .My intellij setting is in the two image .I think I may have some error in settings .
Can you help what' the inconformity with your ide. I want to be a right code sytle .Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-46523173
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin
Posted by concretevitamin <gi...@git.apache.org>.
Github user concretevitamin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1127#discussion_r13989743
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala ---
@@ -245,6 +245,74 @@ case class LeftSemiJoinBNL(
}
}
+
+
+
+/*
+LeftSemiBloomFilterBroadcastJoin
--- End diff --
Mind formatting as the rest of the code base?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...
Posted by YanjieGao <gi...@git.apache.org>.
Github user YanjieGao commented on the pull request:
https://github.com/apache/spark/pull/1127#issuecomment-47869475
Hi all ,I have resolve the conflict . I don't know if this pr has the value to be merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---