You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by navsan <gi...@git.apache.org> on 2016/06/11 03:20:21 UTC
[GitHub] incubator-quickstep pull request #25: Move createNewBlock() out of the criti...
GitHub user navsan opened a pull request:
https://github.com/apache/incubator-quickstep/pull/25
Move createNewBlock() out of the critical section
Move createNewBlock() out of the critical section in BlockPoolInsertDestination::getBlockForInsertion.
This significantly improves performance in queries that write a lot of output.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep fix_getblockforinsertion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/25.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #25
----
commit 688d37aaead71415e70435ea708cff18596df874
Author: Navneet Potti <na...@apache.org>
Date: 2016-06-11T03:18:03Z
Move createNewBlock() out of the critical section in getBlockForInsertion
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #25: Move createNewBlock() out of the criti...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/25#discussion_r66700345
--- Diff: storage/InsertDestination.cpp ---
@@ -339,21 +339,20 @@ void BlockPoolInsertDestination::getPartiallyFilledBlocks(std::vector<MutableBlo
}
MutableBlockReference BlockPoolInsertDestination::getBlockForInsertion() {
- SpinMutexLock lock(mutex_);
- if (available_block_refs_.empty()) {
- if (available_block_ids_.empty()) {
- return createNewBlock();
- } else {
+ {
+ SpinMutexLock lock(mutex_);
+ if (!available_block_refs_.empty()) {
+ MutableBlockReference retval = std::move(available_block_refs_.back());
+ available_block_refs_.pop_back();
+ return retval;
+ } else if (!available_block_ids_.empty()) {
const block_id id = available_block_ids_.back();
available_block_ids_.pop_back();
MutableBlockReference retval = storage_manager_->getBlockMutable(id, relation_);
return retval;
}
- } else {
- MutableBlockReference retval = std::move(available_block_refs_.back());
- available_block_refs_.pop_back();
- return retval;
- }
+ } // release mutex_ lock
--- End diff --
For this comment, we would need one more whitespace indentation.
```
++i; // Comment.
```
On the other hand, we actually don't need this comment, as it is obvious that the scoped lock automatically releases. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
Fix all in this PR. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
Which ones? And should I do that in the same PR?
> On Jun 10, 2016, at 22:26, Zuyu ZHANG <no...@github.com> wrote:
>
> @navsan <https://github.com/navsan> This is great, and you may want to fix the same method of other InsertDestination implementations.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-quickstep/pull/25#issuecomment-225334212>, or mute the thread <https://github.com/notifications/unsubscribe/ACZB6xcITIocrCRPFxUH6p1ByyyLTjwZks5qKir0gaJpZM4IzdWM>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: DO NOT MERGE: Move createNewBlock() out of th...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
@navsan There are other two places need to change, as a part of the same performance issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #25: Move createNewBlock() out of the criti...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/25#discussion_r66700020
--- Diff: storage/InsertDestination.cpp ---
@@ -339,21 +339,21 @@ void BlockPoolInsertDestination::getPartiallyFilledBlocks(std::vector<MutableBlo
}
MutableBlockReference BlockPoolInsertDestination::getBlockForInsertion() {
- SpinMutexLock lock(mutex_);
- if (available_block_refs_.empty()) {
- if (available_block_ids_.empty()) {
- return createNewBlock();
- } else {
+ {
+ SpinMutexLock lock(mutex_);
+ if (!available_block_refs_.empty()) {
+ MutableBlockReference retval = std::move(available_block_refs_.back());
+ available_block_refs_.pop_back();
+ return retval;
+ }
+ else if (!available_block_ids_.empty()) {
--- End diff --
Code style: https://google.github.io/styleguide/cppguide.html#Conditionals
```
if (...) {
} else if (...) {
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
`AlwaysCreateBlockInsertDestination::getBlockForInsertion` and
`PartitionAwareInsertDestination::getBlockForInsertionInPartition`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
Just search the same function name in the same file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #25: DO NOT MERGE: Move createNewBlock() ou...
Posted by navsan <gi...@git.apache.org>.
Github user navsan closed the pull request at:
https://github.com/apache/incubator-quickstep/pull/25
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
@navsan We need to rebase again. The current status that includes changes in 48 files is wired...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: DO NOT MERGE: Move createNewBlock() out of th...
Posted by pateljm <gi...@git.apache.org>.
Github user pateljm commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
@navsan This does look good to me. The logic makes sense as the mutex there is protecting `available_block_refs_` and `available_block_ids_`, so seems ok to move the new memory allocation outside. One request is to write comments in that function to make the intent clear -- this way if there are issues in the future it will be easier to figure things out. A unit test would be awesome too (as you suggested to me earlier today).
Thanks a lot for doing this!
@craig-chasseur -- any comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
Wow! This is really going to get some getting used to.
> On Jun 11, 2016, at 00:06, Zuyu ZHANG <no...@github.com> wrote:
>
> @navsan <https://github.com/navsan> We need to rebase again. The current status that includes changes in 48 files is wired...
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-quickstep/pull/25#issuecomment-225337336>, or mute the thread <https://github.com/notifications/unsubscribe/ACZB60MiVf_Ez3QQsgJxElzVM1DOAuwLks5qKkJYgaJpZM4IzdWM>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: DO NOT MERGE: Move createNewBlock() out of th...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
I made all the changes, per @zuyu's comments. But, in a strange turn of events, performance actually worsened when I moved `createNewBlock()` out of the critical section. We didn't get anywhere in investigating why that might be the case. I'm closing this PR for now and leaving it to future work to reopen it, figure out what's going on, and then merge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #25: Move createNewBlock() out of the criti...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/25#discussion_r66700278
--- Diff: storage/InsertDestination.cpp ---
@@ -339,21 +339,21 @@ void BlockPoolInsertDestination::getPartiallyFilledBlocks(std::vector<MutableBlo
}
MutableBlockReference BlockPoolInsertDestination::getBlockForInsertion() {
- SpinMutexLock lock(mutex_);
- if (available_block_refs_.empty()) {
- if (available_block_ids_.empty()) {
- return createNewBlock();
- } else {
+ {
+ SpinMutexLock lock(mutex_);
+ if (!available_block_refs_.empty()) {
+ MutableBlockReference retval = std::move(available_block_refs_.back());
+ available_block_refs_.pop_back();
+ return retval;
+ }
+ else if (!available_block_ids_.empty()) {
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
@navsan This is great, and you may want to fix the same method of other `InsertDestination` implementations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #25: Move createNewBlock() out of the criti...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/25#discussion_r66700522
--- Diff: storage/InsertDestination.cpp ---
@@ -339,21 +339,20 @@ void BlockPoolInsertDestination::getPartiallyFilledBlocks(std::vector<MutableBlo
}
MutableBlockReference BlockPoolInsertDestination::getBlockForInsertion() {
- SpinMutexLock lock(mutex_);
- if (available_block_refs_.empty()) {
- if (available_block_ids_.empty()) {
- return createNewBlock();
- } else {
+ {
+ SpinMutexLock lock(mutex_);
+ if (!available_block_refs_.empty()) {
+ MutableBlockReference retval = std::move(available_block_refs_.back());
+ available_block_refs_.pop_back();
+ return retval;
+ } else if (!available_block_ids_.empty()) {
const block_id id = available_block_ids_.back();
available_block_ids_.pop_back();
MutableBlockReference retval = storage_manager_->getBlockMutable(id, relation_);
return retval;
}
- } else {
- MutableBlockReference retval = std::move(available_block_refs_.back());
- available_block_refs_.pop_back();
- return retval;
- }
+ } // release mutex_ lock
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
Harshad confirmed that it doesn\u2019t break anything for BlockPoolInsertDestination. I\u2019m not sure about the others.
> On Jun 10, 2016, at 22:39, Zuyu ZHANG <no...@github.com> wrote:
>
> @navsan <https://github.com/navsan> I thought you had confirmed that we don't need the critical section when creating a block. OK, let me check it.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-quickstep/pull/25#issuecomment-225334606>, or mute the thread <https://github.com/notifications/unsubscribe/ACZB62ak34eGQ3QHfBVRArlFQi3JHMELks5qKi4AgaJpZM4IzdWM>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
@navsan I thought you had confirmed that we don't need the critical section when creating a block. OK, let me check it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #25: Move createNewBlock() out of the critical sec...
Posted by navsan <gi...@git.apache.org>.
Github user navsan commented on the issue:
https://github.com/apache/incubator-quickstep/pull/25
I mean, should I do it for all of them, or just some of the more commonly used ones? And can I be sure that createNewBlock() doesn\u2019t need to be in the critical section for all of them? Do we currently use all of them in our code?
Should it be in the same PR or leave that as an issue for later?
Thanks Zuyu!
> On Jun 10, 2016, at 22:35, Zuyu ZHANG <no...@github.com> wrote:
>
> Just search the same function name in the same file.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-quickstep/pull/25#issuecomment-225334475>, or mute the thread <https://github.com/notifications/unsubscribe/ACZB6wzItdfCBZb61vhfSlpKZt9da6pXks5qKi0IgaJpZM4IzdWM>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---