You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by zuyu <gi...@git.apache.org> on 2016/10/10 02:06:59 UTC
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
GitHub user zuyu opened a pull request:
https://github.com/apache/incubator-quickstep/pull/111
Initiated query execution data structure in all Shiftbosses.
Assigned to @hbdeshmukh. Thanks!
This PR initiates all `Shiftboss`es to execute an admitting query, although we limit to only use the single `Shiftboss` for now. The next step is to schedule `WorkOrder`s on any available `Shiftboss`es.
Note that the current implementation is not fault tolerant to `Shiftboss` failures, nor scale-up and down.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep multiple_shiftboss
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/111.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #111
----
commit b0c3746038177c675257884453e0dbd93ba0ad76
Author: Zuyu Zhang <zu...@apache.org>
Date: 2016-10-10T01:44:00Z
Initiated query execution data structure in all Shiftbosses.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-quickstep/pull/111
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/111#discussion_r84499311
--- Diff: query_execution/PolicyEnforcerBase.hpp ---
@@ -163,6 +174,8 @@ class PolicyEnforcerBase {
// Key = query ID, value = QueryManagerBase* for the key query.
std::unordered_map<std::size_t, std::unique_ptr<QueryManagerBase>> admitted_queries_;
+ std::unordered_set<std::size_t> removed_query_ids_;
--- End diff --
Done. Please take another look. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #111: Initiated query execution data structure in ...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/111
@hbdeshmukh What minor clarification?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #111: Initiated query execution data structure in ...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/111
Hi @zuyu
This looks good to me, except one minor clarification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/111#discussion_r83672097
--- Diff: query_execution/PolicyEnforcerBase.hpp ---
@@ -163,6 +174,8 @@ class PolicyEnforcerBase {
// Key = query ID, value = QueryManagerBase* for the key query.
std::unordered_map<std::size_t, std::unique_ptr<QueryManagerBase>> admitted_queries_;
+ std::unordered_set<std::size_t> removed_query_ids_;
--- End diff --
Does this set mean IDs for the queries which have finished their execution? If so, why do we need this list? Isn't that increasing the overhead of metadata management?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/111#discussion_r83880693
--- Diff: query_execution/PolicyEnforcerBase.hpp ---
@@ -163,6 +174,8 @@ class PolicyEnforcerBase {
// Key = query ID, value = QueryManagerBase* for the key query.
std::unordered_map<std::size_t, std::unique_ptr<QueryManagerBase>> admitted_queries_;
+ std::unordered_set<std::size_t> removed_query_ids_;
--- End diff --
Yes, and this set is used in [PolicyEnforcerBase::exitQuery](https://github.com/apache/incubator-quickstep/pull/111/files#diff-c12b9614fe3de39d47a95a80e1980f9cR114) to check `query_id` in [QueryInitiateResponseMessage](https://github.com/apache/incubator-quickstep/pull/111/files#diff-72cc64e73a6199a842b657ce03025a17R168).
It is required if we do above check, as some query (without any rebuilds) may finish earlier (thus being removed from `admitted_queries_`) than when the `QueryInitiateResponseMessage` replies back from a `Shiftboss`.
Finally, as a `TODO`, we could do some `GC` on `removed_query_ids_`, if we finish checks for older removed `query_id`s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #111: Initiated query execution data structure in ...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/111
I guess you have an unpublished reviews?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/111#discussion_r84111075
--- Diff: query_execution/PolicyEnforcerBase.hpp ---
@@ -163,6 +174,8 @@ class PolicyEnforcerBase {
// Key = query ID, value = QueryManagerBase* for the key query.
std::unordered_map<std::size_t, std::unique_ptr<QueryManagerBase>> admitted_queries_;
+ std::unordered_set<std::size_t> removed_query_ids_;
--- End diff --
I may need some more understanding here. Can you remind me the purpose of ``QueryInitiateResponseMessage``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #111: Initiated query execution data structure in ...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/111
Looks good to me, merging. As travis was green before the minor change, I won't wait for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #111: Initiated query execution data structure in ...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/111
Hi @zuyu
I tried GitHub's new review comment system and I guess that's why my comment wasn't visible. Hope you can see it now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #111: Initiated query execution data struct...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/111#discussion_r84497021
--- Diff: query_execution/PolicyEnforcerBase.hpp ---
@@ -163,6 +174,8 @@ class PolicyEnforcerBase {
// Key = query ID, value = QueryManagerBase* for the key query.
std::unordered_map<std::size_t, std::unique_ptr<QueryManagerBase>> admitted_queries_;
+ std::unordered_set<std::size_t> removed_query_ids_;
--- End diff --
Thanks @zuyu
Apologies for being slow to review this PR. Can you please add the TODO about garbage collection on ``removed_query_ids_``, and I will merge this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---