You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by selvaganesang <gi...@git.apache.org> on 2015/09/10 03:12:04 UTC
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
GitHub user selvaganesang opened a pull request:
https://github.com/apache/incubator-trafodion/pull/77
[TRAFODION-9] Non-blocking hbase operation to smoothen the data flow …
…in trafodion engine
The non-blocking Hbase operation for the index maintenance operators is now enabled by
default for IUD, merge and update commands.
The Hbase operations put, checkAndPut and delete operations from the trafodion engine
for the index maintenance is done using Executor Services to make it non-blocking and the
data flow in the trafodion operators can continue up to the point till it needs to pause for completion.
checkAndDelete is never used for index maintenance now.
Also, ensured that index maintenance uses rowset or single row operators when it should.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvaganesang/incubator-trafodion async_operation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/77.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #77
----
commit f6dba67524b8e9bcbef295b2340232ffb878fb58
Author: selvaganesang <se...@esgyn.com>
Date: 2015-09-10T00:51:04Z
[TRAFODION-9] Non-blocking hbase operation to smoothen the data flow in trafodion engine
The non-blocking hbase operation all index maintenance operators is now enabled by
default for IUD, merge and update commands.
The hbase operations put, checkAndPut and delete operations from the trafodion engine
for the index maintenance is done using Executor Services to make it non-blocking and the
data flow in the trafodion operators can continue up to the point till it needs to pause for completion.
checkAndDelete is never used for index maintenance now.
Also, ensured that index maintenance uses rowset or single row operators when it should.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/77#discussion_r39211292
--- Diff: core/sql/executor/ExHbaseIUD.cpp ---
@@ -4193,7 +4188,7 @@ ExWorkProcRetcode ExHbaseAccessSQRowsetTcb::work()
step_ = DONE;
}
break;
- case ASYNC_INSERT_COMPLETE:
+ case ASYNC_OPERATION_COMPLETE:
--- End diff --
I can change it Hans. Also, I wanted to confirm if the stats returned is correct. So, I might do another code push
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/77#discussion_r39180011
--- Diff: core/sql/executor/ExHbaseIUD.cpp ---
@@ -4193,7 +4188,7 @@ ExWorkProcRetcode ExHbaseAccessSQRowsetTcb::work()
step_ = DONE;
}
break;
- case ASYNC_INSERT_COMPLETE:
+ case ASYNC_OPERATION_COMPLETE:
--- End diff --
Would it be better to call this step "COMPLETE_ASYNC_OPERATION"? ASYNC_OPERATION_COMPLETE sounds as if the async operation is already complete, but what it really means is that we need to complete the operation.
Don't want to hold up this change, if you agree then maybe you can rename it in some future checkin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/77
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/77#discussion_r39180445
--- Diff: core/sql/generator/GenRelUpdate.cpp ---
@@ -2652,7 +2656,8 @@ short HbaseInsert::codeGen(Generator *generator)
queue_index downqueuelength = (queue_index)getDefault(GEN_DP2I_SIZE_DOWN);
Int32 numBuffers = getDefault(GEN_DP2I_NUM_BUFFERS);
- if (getInsertType() == Insert::VSBB_INSERT_USER)
+ if (getInsertType() == Insert::VSBB_INSERT_USER &&
+ generator->oltOptInfo()->multipleRowsReturned())
downqueuelength = 400;
--- End diff --
Also, don't queue lengths have to be a power of 2? So, maybe use 512 when we make this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-9] Non-blocking hbase...
Posted by sureshsubbiah <gi...@git.apache.org>.
Github user sureshsubbiah commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/77#discussion_r39172222
--- Diff: core/sql/generator/GenRelUpdate.cpp ---
@@ -2652,7 +2656,8 @@ short HbaseInsert::codeGen(Generator *generator)
queue_index downqueuelength = (queue_index)getDefault(GEN_DP2I_SIZE_DOWN);
Int32 numBuffers = getDefault(GEN_DP2I_NUM_BUFFERS);
- if (getInsertType() == Insert::VSBB_INSERT_USER)
+ if (getInsertType() == Insert::VSBB_INSERT_USER &&
+ generator->oltOptInfo()->multipleRowsReturned())
downqueuelength = 400;
--- End diff --
If at some future point we could change this 400 to a cqd setting or a cardinality based setting or even a constant that will read better. No change is needed now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---