You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/11/09 20:49:10 UTC
[jira] [Commented] (HAWQ-97) Improve performance of Metadata
Versioning by eliminating redundant versioning events
[ https://issues.apache.org/jira/browse/HAWQ-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997241#comment-14997241 ]
ASF GitHub Bot commented on HAWQ-97:
------------------------------------
GitHub user gcaragea opened a pull request:
https://github.com/apache/incubator-hawq/pull/88
HAWQ-97. Optimizing metadata versioning event generation
A DDL command usually generates more than one Nuke Versioning event during its execution. All these nuke events are then pushed to Shared Versioning Queue (SVQ) at commit, and then all of them get propagated to other backends.
This is extra work and it can be optimized. We should push only one Nuke VE instead of several.
Also, when moving messages from the Current Command Invalidation Queue (CVQ) to the Prior Command Invalidation Queue (XVQ), pre-process the event queue to eliminate any events that will have no effect. If a versioning event is followed by a nuke event, it will have no effect.
Moreover, when a nuke event is moved to XVQ, everything prior in XVQ will also have no effect.
Solution is to look for Nuke events in CVQ. Find the last one (in the order of creation), then move all messages from then on to XVQ, overwriting XVQ.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gcaragea/incubator-hawq gcaragea-HAWQ-97
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/88.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #88
----
commit a5baf5b7c9608b6c9e4248b6d158230c1663226d
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-28T19:20:10Z
[HAWQ-97] Optimizing metadata versioning event generation by eliminating a lot of redundant Nuke messages
commit 8aa660386fe003d1d330acb10b6ee66c2fe3d698
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-29T02:34:19Z
[HAWQ-97] Adding unit tests for mdver_dep_translator changes
commit f156a273f62f201524ed0ed81bd356bf0922e88c
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-30T01:09:19Z
[HAWQ-97] Adding unit tests for MdVer_IsRedundantNukeEvent
commit b448c700e99effd208d5e62d2368b301fef36199
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-30T19:11:28Z
[HAWQ-97] Fixed issue with pre-processing invalidation list for MD Ver
commit 4e1dd59d29ceb337aeca8aab91e4ec625db39e59
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-30T19:12:03Z
[HAWQ-97] Adding unit tests for MdVer_PreProcessInvalidMsgs
commit 09b65f047370ca9e4cd0a905dbd22359e16a05f9
Author: George Caragea <gc...@pivotal.io>
Date: 2015-10-31T00:31:49Z
[HAWQ-97] Finished unit tests for MdVer_PreProcessInvalidMsgs
commit a0056992dc324f7ebfcb8791b6643ab07291f77f
Author: George Caragea <gc...@pivotal.io>
Date: 2015-11-09T18:16:33Z
[HAWQ-97] Added function commit to test
commit c95f522e309280ed038ea90edfb087c308317731
Author: George Caragea <gc...@pivotal.io>
Date: 2015-11-09T18:29:18Z
[HAWQ-97] Freeing up discarded chunks from versioning event lists when compacting them
----
> Improve performance of Metadata Versioning by eliminating redundant versioning events
> -------------------------------------------------------------------------------------
>
> Key: HAWQ-97
> URL: https://issues.apache.org/jira/browse/HAWQ-97
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Metadata Versioning
> Reporter: George Caragea
> Assignee: George Caragea
>
> A DDL command usually generates more than one Nuke Versioning event during its execution. All these nuke events are then pushed to Shared Versioning Queue (SVQ) at commit, and then all of them get propagated to other backends.
> This is extra work and it can be optimized. We should push only one Nuke VE instead of several.
> Also, when moving messages from the Current Command Invalidation Queue (CVQ) to the Prior Command Invalidation Queue (XVQ), pre-process the event queue to eliminate any events that will have no effect. If a versioning event is followed by a nuke event, it will have no effect.
> Moreover, when a nuke event is moved to XVQ, everything prior in XVQ will also have no effect.
>
> Solution is to look for Nuke events in CVQ. Find the last one (in the order of creation), then move all messages from then on to XVQ, overwriting XVQ.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)