You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2016/03/24 07:54:05 UTC
[GitHub] spark pull request: [SPARK-14112] [SQL] [WIP] Unique Constraints o...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11930
[SPARK-14112] [SQL] [WIP] Unique Constraints over a Set of AttributeReferences
#### What changes were proposed in this pull request?
This PR is to introduce unique constraints over a set of `AttributeReference`s. Below are just two of use cases:
- Optimizer can remove unnecessary `Distinct` from the plan, as shown in the PR: https://github.com/apache/spark/pull/11854.
- Optimizer can eliminate the no-op `Join` from the plan, as shown in the PR:
https://github.com/apache/spark/pull/9089
We can infer the output uniqueness of the current operator through the uniqueness of the child node's output. The bottom-up propagation rule of unique constraints is
- `Distinct`, `Intersect`, `Except` always return distinct values.
- `Aggregate` has three cases:
1. When its aggregate expressions is identical to its grouping expressions, it is equivalent to `Distinct`. It can always return distinct values.
2. When the child's `outputSet` is subset of its own `outputSet`, it keeps the unique constraints of the child.
3. Otherwise, it does not enforce unique constraints
- UnaryNode `Filter`, `BroadcastHint`, `Sort`, `Window`, `GlobalLimit`, `LocalLimit`, `Sample` and `SubqueryAlias` still keep the unique constraints of the child.
- BinaryNode `Left-semi Join` keep the unique constraints of the left child.
- `Project` keeps the unique constraints of the child if and only if the child's `outputSet` is subset of its own `outputSet`
#### How was this patch tested?
TODO: added a set of test cases for verifying the propagation rules
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark uniquenessConstraints
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11930.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11930
----
commit aea61460ca904883c68cf27e9dacc5ee617efb8e
Author: gatorsmile <ga...@gmail.com>
Date: 2016-03-24T05:58:48Z
introducing Distinct Set
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14112] [SQL] [WIP] Unique Constraints o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11930#issuecomment-200733726
**[Test build #54017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54017/consoleFull)** for PR 11930 at commit [`aea6146`](https://github.com/apache/spark/commit/aea61460ca904883c68cf27e9dacc5ee617efb8e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11930: [SPARK-14112] [SQL] [WIP] Unique Constraints over a Set ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/11930
Sure, let me close it. I will slow it down to make sure my open PRs are less than 20. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14112] [SQL] [WIP] Unique Constraints o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11930#issuecomment-200734437
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11930: [SPARK-14112] [SQL] [WIP] Unique Constraints over...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/11930
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11930: [SPARK-14112] [SQL] [WIP] Unique Constraints over a Set ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/11930
@gatorsmile can you close this? looks like a WIP that hasn't been updated in a long time. You have 16 PRs open at the moment, actually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14112] [SQL] [WIP] Unique Constraints o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11930#issuecomment-200698788
**[Test build #54017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54017/consoleFull)** for PR 11930 at commit [`aea6146`](https://github.com/apache/spark/commit/aea61460ca904883c68cf27e9dacc5ee617efb8e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org