You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by zuyu <gi...@git.apache.org> on 2017/02/05 22:45:25 UTC
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
GitHub user zuyu opened a pull request:
https://github.com/apache/incubator-quickstep/pull/181
Added limited optimizer support for Partitioned Hash Joins.
This PR allows repartition for non-stored, partitioned input relations for hash joins.
Repartition will generate the best partition attribute if both input relations have partitions that do not meet the join condition. Otherwise, repartition on one input relation will reuse the partition attribute from the other that meets the join condition.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep reorder-partitioned-hash-join
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/181.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #181
----
commit fb274c57a80fb98a4ce8318dbb46affb8375e028
Author: Zuyu Zhang <zu...@apache.org>
Date: 2017-01-25T09:49:28Z
Added limited optimizer support for Partitioned Hash Joins.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104584024
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
--- End diff --
Should be good to add a default case here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104878308
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
Now I got your point. I'll do another swap if needed, so that the build side has a less cardinality. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104724405
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has a useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ if (areSamePartitionSchemeHeaders(build_partition_scheme_header, *build_relation,
+ probe_partition_scheme_header, *probe_relation)) {
+ if (cost_model_for_hash_join_->estimateCardinality(build_physical) >
+ cost_model_for_hash_join_->estimateCardinality(probe_physical)) {
+ needs_swap = true;
+ }
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ } else {
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ }
+ break;
+ }
+ }
+ } else if (probe_partition_scheme) {
+ needs_swap = true;
+
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+
+ probe_needs_repartition =
+ (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) ==
+ probe_attribute_ids.end());
+ break;
+ }
+ }
+ } else if (build_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ build_needs_repartition =
+ (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) ==
+ build_attribute_ids.end());
+ break;
+ }
+ }
+ }
+
+ if (needs_swap) {
+ swap(probe_physical, build_physical);
+ swap(probe_attribute_ids, build_attribute_ids);
+ swap(any_probe_attributes_nullable, any_build_attributes_nullable);
+ swap(left_join_attributes, right_join_attributes);
+ swap(probe_operator_info, build_relation_info);
+ swap(probe_relation, build_relation);
+ swap(probe_partition_scheme, build_partition_scheme);
+ swap(probe_needs_repartition, build_needs_repartition);
+ }
+
+ unique_ptr<PartitionScheme> probe_repartition_scheme;
+ if (build_needs_repartition) {
+ const size_t repartition_attr_index =
+ chooseBestRepartitionAttributeIndex(build_relation->getStatistics(), build_attribute_ids);
+ auto build_repartition_scheme_header =
+ make_unique<HashPartitionSchemeHeader>(FLAGS_num_repartitions,
+ build_attribute_ids[repartition_attr_index]);
+ auto build_repartition_scheme = make_unique<PartitionScheme>(build_repartition_scheme_header.release());
+
+ build_partition_scheme = build_repartition_scheme.get();
+
+ if (build_relation_info->isStoredRelation()) {
+ THROW_SQL_ERROR() << "Re-partition for the base build table is not supported";
+ } else {
+ S::InsertDestination *build_insert_destination_proto =
+ query_context_proto_->mutable_insert_destinations(build_relation_info->output_destination_index);
+
+ build_insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::PARTITION_AWARE);
+ build_insert_destination_proto->MutableExtension(S::PartitionAwareInsertDestination::partition_scheme)
+ ->MergeFrom(build_repartition_scheme->getProto());
+
+ CatalogRelation *mutable_build_relation =
+ catalog_database_->getRelationByIdMutable(build_relation->getID());
+ mutable_build_relation->setPartitionScheme(build_repartition_scheme.release());
--- End diff --
May be I missed some of your older PRs, but can you explain how we will physically repartition the relations?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104723411
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has a useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ if (areSamePartitionSchemeHeaders(build_partition_scheme_header, *build_relation,
+ probe_partition_scheme_header, *probe_relation)) {
+ if (cost_model_for_hash_join_->estimateCardinality(build_physical) >
+ cost_model_for_hash_join_->estimateCardinality(probe_physical)) {
+ needs_swap = true;
+ }
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ } else {
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ }
+ break;
+ }
+ }
--- End diff --
I would highly recommend two things for the switch statement:
1. Add several comments in between to describe what's happening. I felt lost while reading it.
2. Moving the switch statement to a separate method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104584264
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
Why do we need to swap here? There's no cardinality check performed here unlike line 863.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104723045
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
I am not sure if I agree. Consider the following counterexample.
Original build table - cardinality 10k, range partitioned.
Original probe table - cardinality 1M, hash partitioned.
By the above logic, you will swap the build and probe relations, thereby building a giant hash table for 1M entries. You are going to repartition the new probe relation anyway, so that cost isn't avoided.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104573225
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
+ }
+ }
+
+ return false;
+}
+
+
+// Note that this method will be deprecated once the partition scheme header
+// supports multiple partition attributes.
+size_t chooseBestRepartitionAttributeIndex(const CatalogRelationStatistics &stats,
+ const vector<attribute_id> &join_attributes) {
+ size_t chose_attr_index = static_cast<size_t>(-1);
--- End diff --
This would assign ``chose_attr_index`` the value of ``MAX_INT``. Why do we want that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104573872
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
+ }
+ }
+
+ return false;
+}
+
+
+// Note that this method will be deprecated once the partition scheme header
+// supports multiple partition attributes.
+size_t chooseBestRepartitionAttributeIndex(const CatalogRelationStatistics &stats,
+ const vector<attribute_id> &join_attributes) {
+ size_t chose_attr_index = static_cast<size_t>(-1);
+ size_t chose_attr_num_distinct_values = 0;
+
+ for (std::size_t i = 0; i < join_attributes.size(); ++i) {
+ const attribute_id attr = join_attributes[i];
+ if (stats.hasNumDistinctValues(attr) &&
+ stats.getNumDistinctValues(attr) > chose_attr_num_distinct_values) {
+ chose_attr_index = i;
+ chose_attr_num_distinct_values = stats.getNumDistinctValues(attr);
+ }
+ }
+
+ return (chose_attr_index != static_cast<size_t>(-1)) ? chose_attr_index : 0;
--- End diff --
I am unable to understand the point of this function. Why do we need to find the "best repartition attribute"? Assuming a join like ``R.a = S.b``, if ``R`` is partitioned on some other attribute ``c``, we would like to repartition it based on ``a``, so the only choice for an attribute on which we should perform the repartitioning is ``a``, isn't it? What's the decision got do do with the CatalogRelation stats and the number of distinct values of the attribute?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104603305
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has an useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
--- End diff --
I intend not to add a default case, so that once a new partition scheme is added, we will have compile errors to force to revisit how the new partition scheme fits in this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104604315
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
+ }
+ }
+
+ return false;
+}
+
+
+// Note that this method will be deprecated once the partition scheme header
+// supports multiple partition attributes.
+size_t chooseBestRepartitionAttributeIndex(const CatalogRelationStatistics &stats,
+ const vector<attribute_id> &join_attributes) {
+ size_t chose_attr_index = static_cast<size_t>(-1);
+ size_t chose_attr_num_distinct_values = 0;
+
+ for (std::size_t i = 0; i < join_attributes.size(); ++i) {
+ const attribute_id attr = join_attributes[i];
+ if (stats.hasNumDistinctValues(attr) &&
+ stats.getNumDistinctValues(attr) > chose_attr_num_distinct_values) {
+ chose_attr_index = i;
+ chose_attr_num_distinct_values = stats.getNumDistinctValues(attr);
+ }
+ }
+
+ return (chose_attr_index != static_cast<size_t>(-1)) ? chose_attr_index : 0;
--- End diff --
In a hash join, if both relations are using range partitions, we need to repartition both relations. Here we use this method to pick up the best single attribute for the new hash repartition. By best, we mean the maximum number of distinct values to minimize the hash collision.
Finally, once we support multiple partition attributes, we don't need this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104584368
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has an useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
--- End diff --
Again, a default case should make the code more extensible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #181: Added limited optimizer support for Partitio...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/181
One case which is currently not considered is the following:
Imagine a join between two relations. The probe side is a result of a selection on a base relation. The probe's base relation is hash partitioned on attribute ``a``. There are two cases:
1. In the projection, the attribute ``a`` is left out, i.e. we don't project it.
2. We project the attribute ``a``.
If you look at the code of ``convertSelection``, the output of the selection has no partitioning scheme. In case two, we can carry forward the partitioning scheme from the base table to the output of the selection. That way, you can avoid the repartitioning step for the probe side.
To summarize, you need to add a check in ``convertSelection`` that can identify case 1 and case 2 mentioned earlier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu closed the pull request at:
https://github.com/apache/incubator-quickstep/pull/181
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104572226
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
--- End diff --
These functions in the anonymous namespace may better be defined in some other source file. This cpp file is getting monstrously big, and any efforts to reduce its size should be welcome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104576949
--- Diff: query_optimizer/ExecutionGenerator.hpp ---
@@ -135,9 +135,11 @@ class ExecutionGenerator {
*/
struct CatalogRelationInfo {
CatalogRelationInfo(const QueryPlan::DAGNodeIndex producer_operator_index_in,
- const CatalogRelation *relation_in)
+ const CatalogRelation *relation_in,
+ const QueryContext::insert_destination_id output_destination_index_in)
--- End diff --
It is used to name an argument in the constructor for a struct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104572424
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
--- End diff --
Can you help me understand how the comparison of two vector references will produce the desired result in this case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104721479
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has an useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
--- End diff --
Are you sure that the compiler will throw an error saying that we didn't handle all the cases in the switch statement? I would recommend a safe approach where we add a default case that throws a fatal error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104720132
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
--- End diff --
I had a discussion with @jianqiao on this issue. He thinks that the partitioning related code should be moved to different functions at least. That way the overall code is easier to read and is also extensible. He's fine with the anonymous namespace arrangement.
An example is all the new code in ``convertHashJoin`` can be moved to separate function(s) and then called from ``convertHashJoin``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by pateljm <gi...@git.apache.org>.
Github user pateljm commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104761783
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
--- End diff --
+1 for the suggestion to move the partition code to a different file. It is (related) but separate module.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104571182
--- Diff: query_optimizer/tests/execution_generator/Partition.test ---
@@ -15,19 +15,35 @@
# specific language governing permissions and limitations
# under the License.
-CREATE TABLE foo (id INT NULL,
- name CHAR(20))
+CREATE TABLE dim (id INT NULL,
--- End diff --
Optional, but it may be easier to read if the relation is named as ``dim_4_hash_partitions``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104577292
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
--- End diff --
The comparison result checks every value in both vectors to ensure whether two have the same range partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104758847
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
I get your point, but the partition number is set by us. So we won't have such insane number, but I could add a TODO here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #181: Added limited optimizer support for Partitio...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/181
Close it temporally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104763709
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has a useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ if (areSamePartitionSchemeHeaders(build_partition_scheme_header, *build_relation,
+ probe_partition_scheme_header, *probe_relation)) {
+ if (cost_model_for_hash_join_->estimateCardinality(build_physical) >
+ cost_model_for_hash_join_->estimateCardinality(probe_physical)) {
+ needs_swap = true;
+ }
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ } else {
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ }
+ break;
+ }
+ }
+ } else if (probe_partition_scheme) {
+ needs_swap = true;
+
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+
+ probe_needs_repartition =
+ (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) ==
+ probe_attribute_ids.end());
+ break;
+ }
+ }
+ } else if (build_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ build_needs_repartition =
+ (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) ==
+ build_attribute_ids.end());
+ break;
+ }
+ }
+ }
+
+ if (needs_swap) {
+ swap(probe_physical, build_physical);
+ swap(probe_attribute_ids, build_attribute_ids);
+ swap(any_probe_attributes_nullable, any_build_attributes_nullable);
+ swap(left_join_attributes, right_join_attributes);
+ swap(probe_operator_info, build_relation_info);
+ swap(probe_relation, build_relation);
+ swap(probe_partition_scheme, build_partition_scheme);
+ swap(probe_needs_repartition, build_needs_repartition);
+ }
+
+ unique_ptr<PartitionScheme> probe_repartition_scheme;
+ if (build_needs_repartition) {
+ const size_t repartition_attr_index =
+ chooseBestRepartitionAttributeIndex(build_relation->getStatistics(), build_attribute_ids);
+ auto build_repartition_scheme_header =
+ make_unique<HashPartitionSchemeHeader>(FLAGS_num_repartitions,
+ build_attribute_ids[repartition_attr_index]);
+ auto build_repartition_scheme = make_unique<PartitionScheme>(build_repartition_scheme_header.release());
+
+ build_partition_scheme = build_repartition_scheme.get();
+
+ if (build_relation_info->isStoredRelation()) {
+ THROW_SQL_ERROR() << "Re-partition for the base build table is not supported";
+ } else {
+ S::InsertDestination *build_insert_destination_proto =
+ query_context_proto_->mutable_insert_destinations(build_relation_info->output_destination_index);
+
+ build_insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::PARTITION_AWARE);
+ build_insert_destination_proto->MutableExtension(S::PartitionAwareInsertDestination::partition_scheme)
+ ->MergeFrom(build_repartition_scheme->getProto());
+
+ CatalogRelation *mutable_build_relation =
+ catalog_database_->getRelationByIdMutable(build_relation->getID());
+ mutable_build_relation->setPartitionScheme(build_repartition_scheme.release());
--- End diff --
We modify the existing `SelectOperator` to have a new `PartitionAwareInsertDestination` for repartition, if the input relation is not a base table. Otherwise, we will create one, but not part of this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104604399
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
+ const CatalogRelationSchema &lhs_scheme,
+ const PartitionSchemeHeader &rhs_partition_header,
+ const CatalogRelationSchema &rhs_scheme) {
+ if (lhs_partition_header.getPartitionType() != rhs_partition_header.getPartitionType()) {
+ return false;
+ }
+
+ if (lhs_partition_header.getNumPartitions() != rhs_partition_header.getNumPartitions()) {
+ return false;
+ }
+
+ // Check whether the underlying types in CatalogAttribute are the same.
+ if (!lhs_scheme.getAttributeById(lhs_partition_header.getPartitionAttributeId())->getType().equals(
+ rhs_scheme.getAttributeById(rhs_partition_header.getPartitionAttributeId())->getType())) {
+ return false;
+ }
+
+ switch (lhs_partition_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kHash:
+ return true;
+ case PartitionSchemeHeader::PartitionType::kRange: {
+ const vector<TypedValue> &lhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(lhs_partition_header).getPartitionRangeBoundaries();
+ const vector<TypedValue> &rhs_ranges =
+ static_cast<const RangePartitionSchemeHeader&>(rhs_partition_header).getPartitionRangeBoundaries();
+
+ return lhs_ranges == rhs_ranges;
+ }
+ }
+
+ return false;
+}
+
+
+// Note that this method will be deprecated once the partition scheme header
+// supports multiple partition attributes.
+size_t chooseBestRepartitionAttributeIndex(const CatalogRelationStatistics &stats,
+ const vector<attribute_id> &join_attributes) {
+ size_t chose_attr_index = static_cast<size_t>(-1);
--- End diff --
That's just an initial value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104570660
--- Diff: query_optimizer/ExecutionGenerator.hpp ---
@@ -135,9 +135,11 @@ class ExecutionGenerator {
*/
struct CatalogRelationInfo {
CatalogRelationInfo(const QueryPlan::DAGNodeIndex producer_operator_index_in,
- const CatalogRelation *relation_in)
+ const CatalogRelation *relation_in,
+ const QueryContext::insert_destination_id output_destination_index_in)
--- End diff --
What do the ``_in`` suffix mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104604523
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -679,13 +688,72 @@ void ExecutionGenerator::convertFilterJoin(const P::FilterJoinPtr &physical_plan
std::piecewise_construct,
std::forward_as_tuple(physical_plan),
std::forward_as_tuple(probe_relation_info->producer_operator_index,
- probe_relation_info->relation));
+ probe_relation_info->relation,
+ probe_relation_info->output_destination_index));
DCHECK(lip_filter_generator_ != nullptr);
lip_filter_generator_->addFilterJoinInfo(physical_plan,
build_filter_operator_index);
}
+namespace {
+
+bool areSamePartitionSchemeHeaders(const PartitionSchemeHeader &lhs_partition_header,
--- End diff --
Unless these functions are useful to other files, we keep them in the same file as the caller.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104583960
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has an useful partition.
--- End diff --
Grammar: an -> a
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104603519
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
In a hash join, if the build relation uses range partition, while the probe uses hash partition, we swap two relations, and then repartition the probe (aka, the original build) relation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #181: Added limited optimizer support for Partitio...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/181
Assigned to @hbdeshmukh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104812928
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
--- End diff --
It's clearly not an insane number. I said **cardinality** and not **number of partitions**. SSB Lineorder table for SF100 has a cardinality of 600M, so 1M is not insane by any means.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #181: Added limited optimizer support for P...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/181#discussion_r104759237
--- Diff: query_optimizer/ExecutionGenerator.cpp ---
@@ -740,6 +805,202 @@ void ExecutionGenerator::convertHashJoin(const P::HashJoinPtr &physical_plan) {
key_types.push_back(&left_attribute_type);
}
+ const CatalogRelationInfo *build_relation_info =
+ findRelationInfoOutputByPhysical(build_physical);
+ const CatalogRelationInfo *probe_operator_info =
+ findRelationInfoOutputByPhysical(probe_physical);
+
+ const CatalogRelation *build_relation = build_relation_info->relation;
+ const CatalogRelation *probe_relation = probe_operator_info->relation;
+
+ // FIXME(quickstep-team): Add support for self-join.
+ if (build_relation == probe_relation) {
+ THROW_SQL_ERROR() << "Self-join is not supported";
+ }
+
+ const PartitionScheme *build_partition_scheme = build_relation->getPartitionScheme();
+ const PartitionScheme *probe_partition_scheme = probe_relation->getPartitionScheme();
+
+ bool build_needs_repartition = false;
+ bool probe_needs_repartition = false;
+ bool needs_swap = false;
+ if (build_partition_scheme && probe_partition_scheme) {
+ const PartitionSchemeHeader &build_partition_scheme_header = build_partition_scheme->getPartitionSchemeHeader();
+ const PartitionSchemeHeader &probe_partition_scheme_header = probe_partition_scheme->getPartitionSchemeHeader();
+
+ switch (build_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ build_needs_repartition = true;
+
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id probe_partition_attr = probe_partition_scheme_header.getPartitionAttributeId();
+ if (find(probe_attribute_ids.begin(), probe_attribute_ids.end(), probe_partition_attr) !=
+ probe_attribute_ids.end()) {
+ needs_swap = true;
+ } else {
+ probe_needs_repartition = true;
+ }
+ break;
+ }
+ }
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
+ const attribute_id build_partition_attr = build_partition_scheme_header.getPartitionAttributeId();
+ if (find(build_attribute_ids.begin(), build_attribute_ids.end(), build_partition_attr) !=
+ build_attribute_ids.end()) {
+ // BuildRelation has an useful partition.
+ switch (probe_partition_scheme_header.getPartitionType()) {
+ case PartitionSchemeHeader::PartitionType::kRange:
+ probe_needs_repartition = true;
+ break;
+ case PartitionSchemeHeader::PartitionType::kHash: {
--- End diff --
Yes, it is `-Wswitch` and then becomes an error for us.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---